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1.0  Summary  and  Goals 

Despite  the  recent  avalanche  of  publicity  on  virtual  reality,  the  state  of  the  an  i 
displays  (HMD)  is  still  at  the  "almost-works”  stage.  Although  extravagant  claims  arc  made  almost 
daily  on  television  and  in  the  popular  press,  there  is  very  link  that  can  be  usefuliy  accomplished 
today  with  virtual  reality  technologies. 

We  at  UNC  have  been  working  since  the  early  1970s  on  aspects  of  these  technolc^ics  and  have 
been  advancing  the  state  of  the  an  by  a  "driving  problem"  approach;  we  let  the  needs  of  selected 
applications  stimulate  the  direction  of  the  technolo^al  tkveiopments  and  then  test  new  results  by 
observing  their  impact  on  solving  the  criginal  applic«ion.  We  are  working  on  three  appbeation 
areas;  molecular  niodcling,  3D  i^ical  imaging,  and  modeling  of  architectunil  interiors. 

We  have  developed  complete  systems  that  include  the  head-mounted  display  device,  the  display 
generation  hardware,  and  a  head  and  hand  tracker.  We  have  access  to  a  force-feedback  ARM  that 
is  owned  by  the  GRIP  j  mject.  We  purchase  components  when  available  and  build  compwicnts 
when  there  was  a  clear  advantage  to  doing  so: 

a)  We  built,  under  separate  major  fun^ng  by  DARPA  and  NSF,  the  display-generation 
hardware  (our  most  recent  machine  is  Pixel-Planes  5.  and  a  new  machine,  PixclFlow.  is  now 
in  the  design  stage). 

b)  We  built  an  optoelectronic  helmet-cracking  system  (the  ceiling  tracker)  that  can  determine 
position  and  arientarion  over  a  large  area  (currently  under  a  10-  x  12-fooi  ceiling).  To  our 
knowledge,  this  is  the  first  demonstrated  large-area  helmet  tracker. 

c)  We  purchase  commercially-available  head-mounted  di^lay  devices  but  continue  to  build 
head-mounted  display  devices  with  see-through  capability. 

Construction  has  begun  on  a  larger  version  of  our  ceiling  tracker.  Covering  four  times  the  area  of 
the  original,  this  will  offer  truly  wide-range  tracking  for  architcaural  walkthrough  applications  and 
see-through  HMD-based  applications,  such  as  ultrasound  examination.  By  developing  an 
algorithm  to  calibrate  beacon  positions,  we  greatly  reduced  the  cost  of  the  new  ceiling.  We  plan  to 
have  the  new  system  operadonal  in  March  1993. 
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1.1  Goals  of  the  Head-Mounted  Display  Project 

•  Demonstrate  the  usefulness  of  the  head-mounted  display  (HMD)  in  real  applications. 

•  Build  a  system  in  which  virtual  objects  appear  to  remain  fixed  as  the  user’s  head  moves. 

•  Improve  the  hardware  subsystems  which  currently  limit  performance  of  HMDs  (trackers,  HMD 
optics  and  displays,  real-time  graphics  generation  engines). 

•  Design  and  implement  software  base  to  support  HMD  functions,  including  model-building 
tools. 
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•  Integrate  visual,  auditory  and  haptic  (force  feedback)  displays  into  a  working  SYSicm 

•  Build  new  input  devices  and  investigate  methods  of  manual  conool  suitable  to  an  immersive 
synthetic  enviroament. 

•  Build  and  investigate  the  uses  and  limitations  of  see-through  HMDs  (optical  and  mergcd-vidco 
designs). 

1.2  Goals  of  the  See*through  Head* Mounted  Display  Subproject 

•  Derive  aadtcsi  a.ihcoretical  model  for  the  registranon  error  m  a  see-ihrtw^  hcad-mounial 
display  (STHMD). 

•  Build  a  low-latency  system  so  that  virtual  objects  appear  to  be  stationary  when  superimposed  on 
the  real  environmenL 

•  Experiment  with  methods  of  registering  real  and  virtual  objects. 

•  Derive  a  method  to  calibrate  a  see-through  head-mounted  display  accurately. 

•  Assess  perception  of  depth  and  sizes  using  a  bench  prototype  STHMD. 

•  Build  and  integrate  a  60-dcgrcc  ficld-of-vicw  STHMD  system. 

•  Design,  build,  and  integrate  a  vitteo  STHMD  using  two  miniamre  video  camexas. 


1.3  Goals  of  the  Tracker  Subproject 

•  Develop  wide-range  trackers  for  head-mounted  displays,  improve  our  existing  tracking  system, 
which  can  track  an  HMD  inside  a  small  room-sized  environment.  For  the  long  term,  explore 
technologies  that  can  track  multiple  users  and  have  unlimited  range  without  requiring 
modification  of  the  environment. 

•  Woik  to  improve  the  other  aspects  of  HMD  tracking:  latency,  speed,  accuracy,  and  resolution. 


1.4  Goals  of  the  Walkthrough  Subproject 

•  Develop  tools  and  techniques  for  building  synthetic  environment  models. 

•  Explore  virtual  environments  throu^  architectural  simulation.  Walking  through  a  building  is  a 
natural  and  intuitive  pitx:ess  in  the  real  world;  the  goal  of  the  Walkthrough  project  is  to  use  this 
intuitiveness  to  study  virtual-world  interaction,  especially  wayfinding  ui  virtual  spaces, 

•  Serve  as  a  driving  problem  for  HlVlD,  tracker,  and  graphics  architecture  research.  Walkthrough 
is  a  real-world  application  that  pushes  the  limits  of  HMD,  tracker,  and  graphics  architwrture 
technologies. 


1.5  Goals  of  the  Virtual-Environment  Ultrasound  Scanning  Subproject 

•  Build  and  operate  a  3D  ultrasound  system  which  acquires  and  displays  3D  volume  data  in  real 
time.  This  design  requires  significant  advances  in  3D  data  acquisition  and  3D  volume  data 
display.  The  former  topic  is  being  investigated  by  a  research  group  at  Duke  University  and  is 
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not  included  in  our  project.  The  lancr  topic  is  Uw  focus  of  Prof.  Henry  Fuchs’  ultnisoufKl 
research  at  UNC-Chapel  Hill.  FunhcnncMe,  accurate  tracking  of  both  the  physician's  head  and 
hand  are  required  to  register  tl«  compu&M-  reconstructed  ultrasound  volume  properly  with  the 
real  world. 


1.6  Goals  of  the  Nanomanipulator  Subproject 

•  Build  a  system  that  a  surface  sciendsi  can  use  to  perform  real-tiroc  modification  experiments  on 
surfaces  under  study  with  a  scanning  tunneling  (STM). 

•  Gain  an  understanding  of  the  basic  processes  for  surface  modification  using  an  STM  by  rtKans 
of  bias  voltage  pulses. 

•  Use  the  system  to  create  quantum  interference  devices,  such  as  quantum  uansistors. 


2.0  Summary  of  Major  Accomplishments 

2.1  Head'Mounted  Display  System 

•  During  this  reporting  period,  we  finished  the  proof  of  concept  of  pre-distonion  of  images  to 
compensate  for  optical  distortion.  We  also  achieved  limited  success  with  pre-distonion  (rf  only 
vertex  locations  of  polygons. 

•  The  first  stages  of  a  Juv-In-Timc  Pixels  system,  which  incorporates  both  knowledge  about  the 
time  of  display  of  a  pixel  and  pre^ctivc  head-tracking  techniques  in  the  comiMitation  of  the 
displayed  pixel  values,  has  been  implemented.  The  intensity  value  displayed  at  each  pixel  is 
based  upon  the  estimattxl  viewpoint  of  the  user  at  the  time  of  its  display. 

•  We  generated  an  animated  sequence  simulating  the  difference  between  images  based  on  /ust-ln- 
Timc  Pixels  and  conventionally  rendered  images.  We  also  verified  that  the  distortion  due  to 
conventional  rendering  techniques  (and  the  corresponding  lack  of  distortion  of  Just-ln-Time 
images)  is  perceptible  in  objects  moving  relative  to  the  user's  viewpoint 

•  A  beam  targeting  tool  to  be  used  for  evaluation  of  the  effect  of  hean  tracking  on  tar^ting 
radiation  treatment  beams  in  ourcer  therapy  has  been  developed  and  is  being  tested  in  user 
studies. 

•  The  team  developed  a  serial  communications  library  so  that  all  serial  trackers  will  be  aWe  to  ron 
over  the  telephone  lines  from  any  of  the  host  machines,  providing  much  greater  flexibility  in 
terms  of  HNID  station  ctxifigurations.  This  is  distinct  from  the  serial  device  library  whk:h 
operates  over  the  network.  The  advantage  to  this  approach  is  that  once  the  phone-line 
connection  is  established,  the  line  is  dedicated  and,  therefore,  impervious  to  network  loads.  This 
enables  us  to  haraile  switching  of  video  sources  in  software  from  any  workstation. 

•  A  3D  color  selection  survey  including  slicing  through  RGB  color  cubes  and  HSV  color  cones 
was  created.  It  could  not  be  detennined  whether  implemented  3D  methods  are  superior  \o  the 
2D  methods  tried  (c.g.,  HSV  sliders)  for  virtual-worids  interaction. 

•  Foot  tracking  became  a  new  area  of  cxpmmentation.  The  user’s  feet  are  tracked  relative  to  the 
body  in  order  to  derive  absolute  position  (given  a  known  stan  point).  A  preliminary  paper 
desciibing  the  system  and  some  of  its  inherent  problems  was  written. 
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The  sound  Utaraiy  for  playing  sounds  in  the  virtual  world  was  rewritten  to  mcrcasc  us 
responsiveness  and  to  enable  the  playing  of  binaural  azimuth- localizable  sounds. 


2.2  See-through  Head-Mounted  Display 

2.2.1  30-Degree  Fleld-of-View  System  Using  Off-the-Shelf  Optics 

During  the  last  reporting  period  we  built  and  integrated  a  first  prototype  of  dur  30-dc|rcc  field-of- 

view  optical  and  mechanical  head-mounted  display  system.  Ehiring  this  reporting  period  we  have 

made  the  following  accomplishments  in  research  using  this  sce-ihrou^  bead- mounted  display; 

•  We  completed  a  first  set  of  {^ychophysical  studies  using  the  optica]  berurh  prototype  s^- 
through  HMD.  Those  includod  size  and  depth  perception  in  vinuaJ  environments,  as  wdl  as  the 
effea  of  mismatch  between  intcrpupillary  distant*  (UPDs)  on  the  user  of  the  HMD  and  on  the 
optics  and  software  setups.  Part  of  this  research  was  funded  under  an  NIH  grant. 

•  A  first  draft  of  a  paper  describing  the  calibration  of  the  system  and  the  results  obtained  has  also 
been  completed. 

•  Dr.  Jannick  RoUand  presented  the  results  of  the  experiments  in  an  invited  talk  at  the  Opdeal 
Society  of  America  by  in  September  1992  (see  Section  5  3). 

•  We  created  a  simple  demonstration  application  to  illustrate  the  lag  problems  inherent  in  see- 
through  head-nwunted  display  <STHDvtD)  systems.  This  STHMD  has  been  used  as  a  testbed  for 
low-latency  rendering  software  and  for  low-latency  tr^rkcr  testing;  work  is  continuing  in  this 
area. 

•  The  Pixel-Planes  5  graphics  software  was  modified  to  handle  the  invened  screen  configuration 
(which  complicated  the  lighting  calculations),  and  an  ”x-ray  vision"  visualization  program  was 
demonstrate  for  attendees  of  tfic  Visualization  in  Biomedical  Computing  conference  held  at 
UNC.  The  demonstration  features  a  computed  tomography  (CT)  dataset  of  a  human  jaw 
superirnposed  onto  a  styrofoam  head  as  a  simulation  of  a  surgical  planning  tool.  The  user  can 
make  tissue-depth  measurements  with  a  virtual  ruler. 

•  The  "x-ray  vision"  demonstration  was  augmented  with  virtual  menus  (from  the  VTK  library 
described  in  the  previous  report)  to  allow  the  user  to  perform  various  operations,  the  most 
impcHtant  of  which  is  the  automatic  registration  of  the  virtual  and  real  objects.  The  user  picks 
“landmarks”  (as  used  by  surgeons  in  current  practice)  on  both  datasets,  and  the  system  uses  a 
principle  components  method  to  align  them.  Currently,  only  the  centroids  are  aligned.  Inn  wwic 
on  implementing  the  rotation  portion  of  the  transformation  should  be  complctal  shortly.  The 
system  was  also  augmented  with  sound  to  give  the  user  feedback  on  menu  selections, 
registration  operations,  and  errors. 

2.2.2  60-Degree  Field-of-View  System  with  Custom  Optics 

•  A  design  review  of  the  optics  of  the  60-dcgrec  ficld-of-vicw  (FOV)  optics  was  conducted  at 
UNC-CT  in  the  presence  of  a  representative  of  Tektronix,  the  company  that  is  providing  the 
two  miniature  color  displays.  The  design  review  showed  that  the  color  plate  u^  in  conjunction 
with  the  CRTs  to  provide  color  needed  to  be  moved  close  to  the  CRT;  this  was  due  mainly  to 
optical  reasons  resulting  from  the  folding  geometry  chosen  for  the  system.  The  color  plate  had 
b«n  nK)ved  after  the  folding  prism  previously  to  facilitate  the  mechanical  assembly  of  the 
system. 

•  The  reoptimization  of  the  tiO-degrce  FOV  optical  system  was  completed  in  Oci'obcr  1992. 
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•  Wc  also  completed  the  desi^  for  the  helmet  to  mount  the  cwnplicatcd  ofmes.  The  has 
adjustments  for  the  intcrpupillary  distance  (IPD)  and  balanced  adjustments  for  the  height  of  the 
optics.  The  design  utilizes  composites  to  reduce  the  mass  of  the  helmet  A  unique  mount  was 
also  designed;  This  mount  creates  an  oil-tight  seal  between  the  lenses  and  the  CRT,  and  it  can 
also  rotate. 

•  Following  the  first  pass  at  the  mechanical  assembly  for  the  60-<fe:gree  FOV  system,  wc 
discovert  that  pressures  exerted  by  the  CRT  from  thermal  expansion  could  result  in  a  loss  in 
optical  image  quality.  Wc  «)lved  this  problem  ^  deciding  to  couple  the  CRTs  to  ilw  opnes  via  a 
thin  film  of  oil  of  0.4mm.  Adding  this  extra  thickness  of  liquid  without  reoptimizing  the  system 
would  put  the  final  mage  out  of  focus,  but  wc  his  was  compensated  fc^r  this  by  having  the  color 
plate  made  slighdy  ininncr. 

•  We  have  received  a  reply  from  Tektronix  that  the  plate  could  be  made  0.01  ”  thinner,  but  ha*  - 
not  yet  received  any  specific  details  about  which  layer  of  the  plate  would  be  mai-  thinner.  .  .is 
informadon  is  necessary  to  proceed  with  the  fine  tuning  of  the  optics. 

2.2.3  Video  See<through  Cameras  with  Custom  Optics 

•  Wc  have  completed  both  the  optical  and  mechanical  design  for  mounting  the  two  miniature  video 
cameras  to  be  used  with  the  Flight  Helmet  from  Virtual  Research.  The  lenses  of  the  video 
cameras  have  been  designed  to  compensate  for  the  optical  distortion  of  tlw  LEEP  optics  used  in 
the  Flight  Helmet,  and  the  FOVs  of  the  two  cameras  were  designed  to  match  the  Flight  Helmet 
FOVs.  The  required  optics  designs  have  been  sent  to  the  selected  manufaaurcr. 

•  Lens  mounts,  which  include  the  adapters  for  placing  the  lenses  off  center  to  the  cameras,  have 
been  designed  for  the  miniature  lenses  . 

•  A  detailed  report  of  the  optics  design  has  been  written  by  graduate  student  Emily  Edwards  and 
Dr.  Jannick  Rolland. 


2.3  Tracking 
Background: 

Wc  first  demonstrated  at  the  ACM  SIGGRAPH  '91  conference  (28  July-2  August  1991)  our 
custom-built  optical  tracking  system,  which  features  a  scalable  work  area  that  currently  measures 
10'  X  12’.  The  sensors  consist  of  four  head-mounted  imaging  devices  that  view  infrared  light- 
emitting  diodes  (LEDs)  mounted  in  standard  size  (2'  x  2’)  suspended  ceiling  panels. 
Photogrammetric  techniques  allow  the  head's  locaticwi  to  be  expressed  as  a  function  of  the  known 
LED  {wsitiems  and  their  projected  images  on  the  sensors.  Eiiscontinuitics  that  occurred  when 
changing  working  sets  of  Li^s  were  ^uced  by  carefully  managing  all  error  sources,  including 
LED  placement  tolerances,  and  by  adopting  an  overdetermined  mathematical  model  for  the 
computation  of  head  position:  space-resection  by  coUincarity.  A  novel  aspect  of  this  system  is  that 
the  range  is  not  limit^  to  the  current  configuration.  By  adding  more  panels  to  the  ceiling  grid,  we 
can  sede  the  system  to  any  desired  room  size,  and  we  have  just  begun  building  a  larger  version  of 
this  system  that  is  four  times  larger  than  die  original.  To  our  knowledge,  this  is  the  first 
demonstrated  scalable  tracking  system  for  HMDs.  [Ward92] 

After  its  introduction  in  the  summer  of  1991,  we  installed  this  optical  tracking  system  in  our 
graphics  laboratory,  where  it  is  now  a  plat  rm  for  further  tracker  research  and  for  running  HMD 
applications.  It  supports  the  Walkthrough  project,  which  retired  their  treadmill  input  ctevice  in 
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favor  of  our  optical  tracker.  Efforts  are  underway  to  integrate  see-through  HMD  capability  with 

this  tracker  to  help  the  Ultrasound  group. 

Achievements  during  this  reporting  period: 

•  Construction  of  the  new  ceiling  has  begun.  This  is  a  much  larger  version  of  the  existing  ceiling 
tracker.  Spanning  about  16'  x  30’,  it  covers  four  times  as  much  area  as  the  existing  system. 
Circuit  boids  were  designed  for  the  panels  and  supporting  electronics.  We  are  now  testing 
prototypes  of  these  boards.  On  25  January  1993.  construction  crews  began  tearing  down  office 
wails  to  build  the  new  TrackcrAJltrasound  laboratory  where  the  new  ceding  tracker  will  reside. 

•  We  refined  our  calibration  algorithm  for  measuring  the  locations  of  the  ceiling  tracker's  LEDs. 
To  demonstrate  it,  we  physically  moved  some  panels  in  the  existing  ceiling  ai^  then  measured 
the  new  LED  locations.  This  work  was  done  in  conjunction  with  John  F.  Hughes  of  Brown 
University,  and  we  wrote  and  submitted  a  paper  detailing  this  method  {see  Section  5.2).  The 
ability  to  measure  LED  locations  in  the  ceiling  means  we  will  no  longw  have  to  carefully  align 
the  LED  positions  to  a  known  grid.  This  greatly  reduces  the  expense  needed  to  build  larger 
ceding. 

•  We  implemented  a  simple  prediction  method  on  the  ceiling  HMD.  Accurate  prcdicticwi  requires 
accurate  knowledge  of  time  in  the  system.  To  obtain  this,  we  bypass  UNIX  entirely  and 
directly  inject  the  ceiling  tracker  data  into  our  graphics  engine,  Pixel-Planes  5.  Clocks  on  the 
tracker  computers  and  on  Pixel-Planes  5  are  synchronized  to  within  a  milliscctmd  and  arc 
resynchronized  every  two  to  three  minutes  to  compensate  for  drift.  Mewe  advanced  prediedon 
techniques  wdl  be  explored  in  the  next  reporting  period. 

•  Gyros  and  accelerometers  have  been  acquired  for  use  in  aiding  predictive  tracking  and 
eventually  in  experimenting  with  inertial  tracking.  We  tested  them  individually  with  a  PC-based 
A/D  card,  buUt  mechanical  mounts  for  the  sensors,  and  are  in  the  process  of  integrating  them 
with  the  ceiling  HMD. 

•  A  video  camera  is  mounted  above  the  right  eye  of  the  ceiling  HMD  to  provide  a  primitive  video 
see-through  capability  in  one  eye.  This  will  eventually  support  evaluation  of  prediction  methods 
and  ultrasound  applications. 

•  We  built  a  mount  that  holds  an  ultrasound  transducer  and  three  optical  sensors.  With  this,  the 
ceiling  can  track  the  position  and  orientation  of  the  ultrasonic  sensor,  which  will  permit  the 
Ultrasound  group  to  collect  data  without  worrying  about  magnetic  distortions  to  which 
Polhemus  and  Ascension  trackers  are  vulnerable. 

•  We  have  reduced  the  jitter  that  occurred  when  one  of  the  camera  views  moved  off  the  ceiling. 
This  was  accomplished  by  modifying  the  code  to  assign  weights  to  each  LED  in  the  ceiling, 
with  progressively  smaller  weights  being  assigned  to  the  LEDs  closer  to  the  edge  of  the  cteiling. 
Use  of  these  weights  gradually  reduces  the  importance  of  a  view  as  it  moves  closer  to  an  edge, 
thus  reducing  the  jitter  caused  by  its  sudden  disappearance  when  it  falls  off  the  edge. 

•  We  pierformed  some  tests  to  estimate  the  magnitude  of  the  distortion  in  our  Polhemus  and 
Ascension  trackers.  Further  end-to-end  timing  studies  were  also  performed  on  these  and  other 
trackers.  The  results  arc  being  recorded  in  a  technical  report. 

•  We  investigated  sources  of  noise  in  an  attempt  to  get  cleaner  and  more  accurate  signals  from  our 
optical  sensors.  The  investigation  did  not  identify  any  large  error  sources. 

•  A  safety  mechanism  was  developed  and  installed  in  the  standard  software  to  prevent  us  fixjm 
firing  the  LEDs  more  often  than  their  specified  limits  permiL  We  have  notict^  that  about  four 
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LEDs  in  the  existing  ceiling  appear  to  be  dead,  so  this  new  mechanism  should  protect  us  firexn 
burning  out  more  LEDs. 

•  Preliminary  studies  have  been  done  on  the  feasibility  of  making  a  wireless  versiem  of  the  ceiling 
HMD  so  that  the  user  will  have  no  cable  to  drag. 

•  We  submitted  a  short  article  describing  the  ceiling  cracker  and  the  rexjuirements  tkmaxxlcd  of 
trackers  in  see-through  systems  to  Communicarions  of  the  ACM,  for  a  special  issue  cm 
augmented  reality  systems. 


2.4  Interactive  Building  Walkthrough 

•  We  have  reached  the  stage  where  interactive  radiosity  is  functional  and  can  be  integrated  easily 
into  existing  applications  to  provide  highly  realistic  lighting  simulations  in  near-real  time. 

•  We  have  moved  automatic  model  mesh-generation  in  the  Virtus  WaikThrt>ugh-to-Pixcl-Planes  5 
modeling  pipeline  onto  the  graphics  processors,  allowing  it  to  wexk  more  tightly  with  interactive 
radiosity. 

•  Significant  process  has  been  made  in  building  a  highly  detailed  architeccural  nrxxkl  to  serve  as  a 
testbed  for  user  studies  cm  the  effectiveness  of  different  types  of  detail  in  architectural 
environments  and  on  wayfinding  methods. 

•  We  have  continued  running  user  studies  with  Walkthrough  under  die  ceiling  tracker  to  study  the 
effectiveness  and  shoncomings  of  the  existing  system. 


2.5  Virtual-Environment  Ultrasound  Scanning  System 

•  We  implemented  a  system  to  enhance  visual  depth  cues  by  displaying  a  virtual  “pit”  within  the 
patient.  The  wails  of  the  pit  are  opaque,  which  allows  the  physician  to  judge  better  where  the 
ultrasound  volume  lies  within  the  paticnL 

•  The  implementation  of  a  software  system  to  reconstruct  a  scries  of  2D  ultrasound  images  and 
their  legations  and  orientations  into  a  3D  regular  volume  has  been  completed.  This  is  deme  off¬ 
line. 

•  The  Vlib  virtual  world  management  capabilities  arc  now  integrated  into  WEVOL  to  allow 
volume  rendering  in  a  virtual  environment 

•  We  integrated  ultrasound  image  acquisition,  resampling,  and  volume  rendering  into  a  virtual 
environment  to  create  one  large  interactive  ultrasound  virtual-reality  applicaricm. 

•  A  system  for  calibrating  both  the  ultrasound  transducer  and  the  sec-through  video  for  use  under 
the  optical  ceiling  tracker  was  developed.  The  calibration  procedure  is  independent  of  the  type 
of  tracking  or  ]»sition/orientation  of  the  transducer,  allowing  the  system  to  functiem  under 
changing  conditions.  Further  work  is  necessary,  but  the  system  shows  positive  preliminary 
results. 

•  Dr.  Vem  Katz  of  the  UNC  Hospitals'  Department  of  Obstetrics  and  Gynecology  us«i  our 
volume-rendering  ultrasound  system  to  scan  a  fetus  inside  a  pregnant  woman. 

•  We  designed  and  simulated  in  software  a  real-time  video  frame  grabber  for  the  Fhxcl-Plancs  5 
high-speed  ring.  The  design  for  the  board  has  gone  to  the  Microelectronics  Center  of  North 
CioUna  for  fabrication. 
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General  Electric  Ultrasound  Systems  has  made  a  generous  loan  to  us  of  a  new  ultrasound 
scanner.  This  scanner  has  increased  the  resolution  of  our  ultrasound  images.  FurthcrmcM'e.  it 
has  many  advanced  features  lacking  on  our  previous  machine. 


2.6  The  Nanomanipulator:  An  Atomic>$cale  Teleoperator 

•  We  have  interfaced  the  system  with  an  HP8131A  pulse  gencratOT  to  provide  bias  pulses  under 
user  control.  We  have  used  pulses  to  place  mounds  of  gold  onto  a  surface  at  locations  specified 
by  the  user  in  real  time,  using  our  manipulator  ARM. 

•  We  have  characterized  the  elements  of  the  feedback  loc^  using  a  network  analyzer.  With  this 
information,  we  designed  a  new  feedback  control  circuit  to  optimize  system  performance. 


3.0  Expected  Milestones  during  the  Next  12  Months 

3.1  Head-Mounted  Display  System 

•  Pre-distOTt  images  more  accurately  by  dicing  large  polygons  and  nxtving  polygon  vertices. 

•  Continue  development  of  the  Just-In-Time  Pixels  rcndcrcr.  Primary  goals  will  be  the 
incorporation  and  fine  tuning  of  predictive  tracking  techniques  and  the  inclusion  of  the 
ability  to  handle  more  complex  scene  geometries. 

•  Complete  a  study  of  the  effect  of  head-mounted  display  on  targeting  radiation  therapy 
beams.  Practicing  radiation  oncologists,  dosimemsts,  and  radiation  physicists  are 
currently  being  recruited  for  participation  in  a  user  study.  This  study  will  make 
subjective  and  objective  comparison  between  treatment  beam  configurations  designed 
with  a  head-tracked  navigation  mode  and  those  designed  without  head  tracking. 

•  Write  and  publish  a  paper  describing  the  essential  coordinate  systems  and 
transformations  for  a  head-mounted  display  system. 


3.2  See-through  Head-Mounted  Display 

3.2.1  30-Degree  Field-of-View  System  Using  Off-the-Shelf  Optics 

•  Write  the  final  draft  of  the  paper  describing  the  calibration  of  the  system  and  psychophysical 
data  obtained. 

•  Write  a  research  proposal  to  obtain  further  funding  for  this  research. 

•  Procure  and  calibrate  CCD  cameras  to  measure  registration  error  for  the  30-degree  FOV  system. 

•  Build  measurement  setup  for  analyzing  registration  error  for  a  simple  system. 

•  Derive  a  model  for  registration  error  as  a  function  of  error  sources  (distortion,  tracker  error, 
alignment  error,  etc.). 

•  Measure  registration  error  for  subsets  of  the  STHMD  system  of  ever-increasing  size,  and 
compare  them  with  error  predicted  by  the  model. 

•  Complete  the  calibration  procedure  for  the  see-through  HMD  system. 
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3.2.2  60>Degree  Field-of-View  System  with  Custom  Optics 

•  Complete  the  fine  tuning  of  the  optics  design  once  exact  numbers  arc  provided  by  Tektronix. 

•  Complete  the  mechanical  design  of  the  system. 

•  Have  the  optics  and  the  mechanical  parts  for  the  assembly  manufactured. 

•  Assemble  the  first  prototype  of  the  system. 

3.2.3  Video  See-through  Cameras  with  Custom  Optics 

•  Investigate  alternatives  for  mounting  the  cameras  on  the  helmet 

•  Assemble  the  cameras  after  receiving  the  lenses  and  the  mechanical  parts. 

•  Evaluate  the  system. 


3.3  Tracking 

•  Finish  construction  of  the  new  ceiling  and  get  it  running.  This  includes  hardware  setup  and 
software  modifications.  Run  our  calibration  method  to  measure  the  LED  locations  in  the  new 
ceiling.  Successful  measurement  of  LED  locations  will  control  discontinuities  in  the  computed 
position  and  orientations  reported  by  our  tracker.  Discover  how  much  this  increased  range  of 
motion  helps  in  certain  applications. 

•  Mount  the  gyros  and  accelerometers  on  the  ceiling  HMD,  and  use  them  to  suppon  advanced 
prediction  methods.  Develop,  demonstrate,  and  evaluate  prediction  methods  with  both  objective 
measures  and  user  studies. 

•  Investigate  better  math  techniques  for  computing  the  position  and  orientation  of  the  head  with 
our  equipment  Our  existing  method,  space-resection  by  coUincarity,  is  an  iterative  method  that 
does  not  always  converge  to  the  correct  solution  in  difficult  conditions.  Other  methods  should 
be  explored.  One  particular  alternative  flashes  only  one  LED  per  update,  using  an  incremental 
Kalman  filter  to  estimate  the  new  position  and  orientation.  It  has  the  potential  to  improve 
significandy  the  accuracy,  update  rate,  and  lag  of  this  tracker. 

•  In  order  to  support  exploradon  of  alternate  methods,  revise  the  ceiling  tracker  system  software. 
This  will  add  the  ability  to  record  user  modon  with  accurate  dmestamps  and  low-level 
informadon,  such  as  individual  sensor  photocoordinates.  The  programs  which  control  the 
hardware  already  offer  dmestamp  support  that  is  accurate  to  about  30  microseconds.  By 
recording  data  runs,  we  can  apply  altemadve  mathemadcal  methods  to  the  data  to  compute 
position  and  oiientations  off-line. 

•  Integrate  the  see-through  capability  for  at  least  one  eye  on  the  ceiling  HMD  in  a  more  rigorous 
manner  than  the  current  implementation.  Build  a  rigid  mount  to  hold  the  camera,  and  wenk  on 
actually  achieving  reasonable  registration  of  real  and  virtual  objects.  See-through  cjq)ability 
makes  latency  much  more  visible,  aiding  evaluation  of  delay  compensation  techniques. 

•  In  the  long  run,  explore  technologies  for  unlimited  range  tracking  in  unstructured  environments. 
An  area  that  needs  exploration  is  relative-mode  tracking,  which  measures  only  the  relative 
differences  in  position  and  orientation  as  the  user  moves  and  integrates  these  ifferences  over 
time  to  recover  the  head's  location.  Inertial  systems  are  an  example  of  such  a  tracker.  The  main 
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problem  with  relative-mode  trackers  is  drift,  since  repeated  integration  accumulates  error  over 
time.  A  hybrid  inertial-optical  tracker  could  prove  to  be  an  effective  system.  Occasional  fixes 
from  the  optical  tracker  will  prevent  drift  from  accumulating  endlessly,  and  the  inertial  units  will 
allow  much  greater  fi:eedom  of  head  orientation  and  immunity  to  environmental  distortions. 
Such  a  system  could  dramatically  reduce  the  number  of  LEDs  required  or,  for  an  equivalent 
cost,  greatly  expand  the  range  of  the  tracking  system. 


3.4  Interactive  Building  Walkthrough 

•  Integrate  more  advanced  versions  of  the  Virtus  WalkThrough  model-building  into  the  system, 
with  support  for  textures,  model  partitioning,  more  complex  radiosity  emitters,  and  the 
replacement  of  model  parts  with  objects  from  our  model  libraries. 

•  Refine  the  current  interactive  radiosity  system  to  increase  the  accuracy  of  the  lighting  model,  to 
lower  the  lag  time  required  to  get  an  initial  solution  to  the  lighting  model,  and  to  smooth  the 
transitions  as  more  accurate  solutions  are  obtained. 

•  Incorporate  adaptive  and  dynamic  meshing  into  the  automatic  model  mesh-generation  to  provide 
mesh  refinement  and  to  allow  the  user  to  change  the  geometry  of  the  scene. 

•  Implement  an  automatic  model  partitioning  scheme  by  using  hierarchy  and  containment 
information  exported  from  Virtus  Walkthrough. 

•  Conduct  additional  user  studies. 


3.5  Virtuai'Environment  Ultrasound  Scanning  System 

•  Build  and  integrate  a  real-time  video  frame  grabber  to  allow  ultrasound  image  acquisition  at  a 
rate  of  30  Hz. 

•  Improve  calibration  techniques  for  merging  computer-generated  and  real-world  images  in  a 
virtual  environment.  The  calibration  proc^ure  previously  discussed  is  being  designed  to  solve 
both  video  and  transducer  calibration  problems. 

•  Track  the  physician’s  head  and  hand  under  the  optical  ceiling  tracker,  as  opposed  to  using  a 
magnetic  tracker  like  a  Polhemus  or  Ascension  tracker. 

•  Improve  the  quality  of  the  image  rendering. 

•  Conduct  an  experiment  in  which  all  resampling  and  rendering  is  done  off-line  to  allow  the 
maximum  image  resolution.  This  will  also  drastically  cut  the  perceived  lag  in  the  system. 

•  Conduct  experimental  trials  of  the  improved  system  with  an  Ob-Gyn  physician,  an  ultrasound 
technician,  and  several  pregnant  subjects. 


3.6  The  Nanomanipulator:  An  Atomic*scale  Teleoperator 

♦  Modify  the  STM  so  that  the  wave  form  of  the  tutmeling  current  resulting  from  voltage  pulses 
can  be  observed  and  recorded. 

•  Work  with  our  collaborator  in  this  research.  Professor  Stanley  Wr  liams  of  the  UCLA 
Department  of  Chemistty,  to  conduct  other  experiments  possible  with  these  new  facilities. 
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4.0  Discussion  of  Research 
4.1  Head-Mounted  Display  System 
Hardware 

The  current  HMD  system  is  used  by  many  projects  and  subprojects  in  a  variety  of  configtirations. 
The  following  is  an  overview  of  the  current  system; 


Figure  1.  Overview  of  UNC  Head-Mounted  Display  System. 


A  description  of  each  component  follows: 

•  Host  computer  A  Sun  4,  which  runs  the  host  side  of  the  application  and  interfaces  to  the 
graphics  engine. 

•  Graphics  Engine:  Pixel-Planes  5,  a  message-passing  multicomputer  capable  of  d*  rwing  more 
than  two  million  Phong-shaded,  Z-buffered  mangles  per  second.  Multiple  applications  can  run 
on  the  system  at  the  same  time  due  to  its  ability  to  be  split  into  several  subsystems. 

•  Magnetic  trackers:  We  use  Fastrak,  3SPACE,  and  Ascension  magnetic  trackers. 

•  Ceiling  tracker  ascribed  in  Section  4.3. 

•  Force-feedbacK  ARM:  Simulates  forces  in  the  virtual  world.  It  has  six  degrees  of  freedom  {3 
forces  and  3  torques)  and  uses  computer-controlled  servo  motors. 

•  Manual  input  devices:  The  newest  input  device  has  a  natural  handgrip  shape  with  five  buttons 
for  user  input  We  also  have  holloweid-out  billiard  balls  with  3SPACE  or  Fastrak  sensors 
mounted  inside,  and  switches  on  the  outside  for  signaling  user  actions. 
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•  Sound:  Controlled  by  Macintosh;  plays  sounds  under  application  control  either  in  earphones 
attached  to  HMD,  or  from  separate  speakers.  The  sound  Macintosh  can  be  controUttl  b>  any 
machine  on  our  network  transparently  using  sdilib. 

In  general,  an  application  can  am  with  almost  any  subset  of  the  above  cquipii^ni 
Software 


Figure  2.  Overview  of  UNC  Virtual-Worlds  Software. 

(Dashed  lines  indicate  application  calls;  solid  lines  indicate  inter-library  calls.) 


The  virtual- worlds  base  software  is  fairly  complete  but  will  continue  to  expand  to  be  more  flexible 
and  powerful  and  to  £;cominodatc  new  hardware.  Most  of  the  libraries  suppOTt  multiple 
technologies  transparently,  allowing  application  code  to  switch  easily  between  different  pieces  of 
hardware  by  changing  only  an  environment  variable.  While  efforts  at  improving  these  libraries 
will  continue,  we  will  wotk  to  add  new  capabilities  and  develop  new  applications.  The  software 
goals  are  listed  in  with  the  other  twelve-month  goals  for  the  jmojcct  in  Section  3. 1 . 


4.2  See-through  Head-Mounted  Display 

ITic  search  for  the  perfect  display  device  for  HMDs  is  ongoing.  While  most  commercial  research 
is  aimed  at  large  screen  TVs,  projection  systems.  HDTV,  and  largc-scrccn  flat  panels,  only  a  very 
small  portion  of  current  research  deals  with  the  development  of  small,  high-tesolution  monitors. 
We  continue  the  evaluation  of  different  technologies  arid  will  incorporate  those  which  cmi  extend 
the  usefulness  and  availability  of  HMDs. 

Research,  both  in  hardware  and  human  factors,  will  be  extended  to  different  types  of  HMDs. 
These  include  die  standard,  total-immersion  variety  and  see-through  (video  ai^  optical)  HMDs. 

Reducing  the  weight  of  HMDs  is  also  a  large  area  of  concent^tion  of  research  cfTc»t.  Due  to  the 
highly  constrained  60-degree  FOV,  sec-through  HMD  specification,  lighter  weight  in  this  design 
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has  been  sacrificed  lo  some  extent  in  order  to  achieve  a  higher  image  quality  and  an  ideal  geometry 
Indeed,  the  geometry  is  thought  to  be  as  imponant  to  the  comfon  of  the  HMD  user  as  the  weight 
itself.  The  first  prototype  will  be  built  using  conventional  optics  since  the  technology  of  making 
lenses  from  glass  or  plastic  is  well  understood.  An  exciting  and  promising  way  to  reduce  the 
system's  weight  and  its  complexity,  however,  is  to  explore  the  usefulness  of  combining  classical 
optics  with  binary  optics.  This  seems  the  logical  step  to  take  in  the  design  of  the  off-axis  60- 
degree  FOV  system.  Other  general  weight-saving  tncks,  such  as  composite  headgear,  plastic  lens 
mounts,  and  wireless  video  transmission,  arc  under  investigation. 

The  design  of  a  video  see-through  system  using  miniature  cameras  with  the  Flight  Hcimci  from 
Virtual  Research  was  completed.  The  main  characicnsiics  of  this  system  are:  the  cameras'  FOVs 
match  the  FOVs  of  the  Flight  Helmet,  the  chosen  amount  of  distortion  of  the  cameras’  lenses  can 
compensate  exactly  far  the  strong  distonion  of  the  LEEP  optics  used  in  the  Right  Helmet,  and  the 
offsets  of  the  CCD  detectors  arc  used  in  the  cameras  to  match  the  centers  of  distcrtion  of  the 
cameras  with  those  of  the  LEEP  optics.  For  example,  the  LCDs  within  the  Right  Helmet  arc  offset 
with  respect  to  the  LEEP  optics  by  6.5  mm  to  accommodate  for  mechanical  assembly  of  the  Right 
Helmet  We  are  currently  woridng  on  how  to  mount  die  cameras  on  the  Flight  Helmet  to  minimize 
any  magnification  of  the  real  world  with  respect  to  the  virtual  world.  To  achieve  a  magnification  of 
1,  the  canwras  would  have  to  be  mounted  as  if  their  viewpoint  were  the  eyes  of  the  user's 
eyepoints.  This  is  not  feasible,  as  explained  in  the  paper  written  by  Edwards  and  Rolland,  but  the 
solution  adopted  may  yet  be  adequate.  One  of  our  goals  is  to  test  the  system  for  the  adequacy  of 
our  chosen  solution.  If  the  test  results  are  negative,  the  next  step  would  be  to  redesign  a  complete 
heliiKt  and  video  camera  assembly. 


Figure  3.  Optical  Layout  of  the  Folded  See-through  60-Degrec  Ficld-of-Vicw  HMD. 
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The  goal  of  the  errOT-raodcl  research  is  to  deiennine  the  registration  error  between  a  real  reject  and 
a  virtual  object  that  are  supposed  to  be  in  the  same  location.  This  error  is  a  function  ^..^f  viewer 
position  and  orientation,  as  well  as  the  head’s  angular  and  linear  velocities.  The  theoretical  oror 
model  will  be  derived  first,  and  then  measurements  on  a  real  system  will  be  made  to  check  the 
mcxtel’s  ability  to  predict  the  registration  error. 

Since  the  registration  is  a  time-varying  visual  phenottienon,  small  cameras  posititmed  where  the 
user’s  eyes  normally  would  be  will  be  used  to  record  the  discrepancy  between  tlte  real  points  and 
their  virtual  counterparts.  We  will  first  build  a  simple  system,  ^rivc  its  theoretical  error,  and  then 
measure  its  real  error.  The  system  wUl  then  be  augmented  to  make  it  more  like  a  con^leic  STHMD 
system,  and  the  process  will  be  repeated.  In  the  end,  we  should  have  a  complete  model  that 
predicts  the  behavior  of  our  existing  system  as  a  function  of  time,  position,  orientation,  and 
velocity.  This  model  will  also  include  a  sensitivity  analysis,  which  will  show  how  an  citot  in  any 
stage  of  the  system  will  be  propagated  through  the  system  to  affect  the  final  registration  error.  We 
will  focus  our  efforts  on  attaclang  the  most  important  errors  first. 


4.3  Tracking 

It  is  well  known  that  trackers  for  HMDs  require  high  resolution,  high  update  rate,  and  low  latency 
Range,  however,  has  not  received  similar  emphasis.  Many  virtual  worlds,  such  as  architectural 
walkthroughs,  would  benefit  from  more  freedom  of  movement  than  is  provided  by  existing 
trackers.  Wide-range  trackers  would  allow  larger  areas  to  be  explored  naturally,  chi  foot,  reducing 
the  need  to  resort  to  techniques  such  as  "flying"  or  walking  on  treadmills.  Also,  such  techniques 
of  extending  range  work  only  with  closed-view  HMDs  that  completely  obscure  reality.  With  see- 
through  HMDs,  which  we  are  attempting  to  build  for  medical  applications,  the  user’s  visual 
connection  with  reality  is  intact,  and  hybrid  applications  in  which  physical  objects  and  con^utcr- 
generated  images  coexist  are  possible.  In  this  situation,  flying  through  the  model  is  meaningless. 
The  model  is  registered  to  the  physical  world  and  one's  relationship  to  both  must  change 
simultaneously. 

Our  optically-based  ceiling  tracker  gives  us  rtxHn-sized  tracking  capability  today  and  is  a  vehicle 
for  tracker  and  HMD  application  research.  The  advent  of  a  much  largCT  ceiling  will  give  us  truly 
long-range  capability  which  we  hope  will  improve  exploration  of  architectural  walkthroughs  and 
provide  a  framework  for  see-through  HMD  applications.  In  the  long  term,  however,  we  would 
like  unlimited  range  tracking  in  unmodified  environments.  No  technology  today  can  provide  this, 
but  a  hybrid  inertial-optical  system  might  prove  to  be  an  effective  solution  in  the  next  few  years. 

Larger  Ceiling  Tracker 

One  of  the  novel  aspects  of  the  ceiling  tracker  is  that  it  can  be  scaled  to  any  desired  room  size.  1116 
current  10'  x  12'  system  is  about  the  size  of  a  small  office  or  a  small  dOTinitory  room.  While  that 
is  large  enough  to  provide  some  freedom  of  movement,  it  reaUy  is  not  very  big.  For  example, 
some  users  find  it  frustrating  to  explore  a  virtual  kitchen  that  is  larger  in  area  than  die  tracking 
system  can  cover.  Thus,  we  are  in  the  process  of  building  a  new  ceiling  that  covers  approximately 
16'  X  30',  or  about  four  times  the  area  of  the  existing  system. 

The  existing  ceiling  was  built  to  exacting  tolerances  to  maximize  the  chances  of  success  of  the 
original  system.  If  the  locations  of  the  beacons  in  the  ceiling  are  misplaced  by  several  millimeters, 
noticeable  errors  can  be  observed  in  the  computed  position  and  orientations.  Each  panel  was 
stretched  flat,  suspended  from  an  external  superstructure,  and  carefully  positioned  with  a  laser 
level.  The  panels  themselves  are  based  on  a  sparsely  populated  circuit  board  that  measures  almost 
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2'  X  2'.  These  measures  keep  the  LEDs  to  their  specified  locations  within  a  tolerance  of  1-2  mm, 
but  at  significant  cost. 

Since  the  new  ceiling  has  four  times  as  many  panels  as  the  original,  several  measures  weir  taken  to 
reduce  the  cost.  First,  two  new  circuit  boards  were  designed  to  replace  the  original  2’  x  2’  board. 
One  board,  of  which  there  are  four  per  panel,  is  a  long  strip  to  hold  the  LEDs,  and  the  second 
holds  the  controlling  circuitry  for  the  panel.  Since  the  area  of  the  two  boards  is  much  smaller  than 
that  of  the  original  circuit  board,  they  are  considerably  cheaper.  Second,  instead  of  trying  to  fix  the 
LEDs  to  a  predetermined  grid,  we  place  them  in  a  ceiling  and  then  measure  where  they  are  using 
calibration  techniques  (see  the  next  section).  This  means  that  the  tolerances  on  panel  construction 
can  be  much  looser  than  they  were  in  the  original  panels.  Third,  we  are  dropping  the  new  panels 
into  standard  2'  x  2'  ceiling  grids.  This  is  made  possible  by  a  much  lighter  panel  design, 
adherence  to  UL  and  fire  codes,  and  our  ability  to  measure  the  locations  of  the  LEDs.  This  means 
that  we  can  dispense  v  ith  an  external  ceilin?  superstructure  that  holds  the  panels  in  precise 
locations.  To  a  casuar  observer,  the  new  cciiing  tracker  should  look  indistinguishable  fiom  any 
other  ceiling  in  our  building,  thus  reducing  the  intrusiveness  of  the  tracker. 

Calibration  of  LED  Locations 

In  collaboration  with  Professor  John  F.  Hughes  of  Brown  University  and  Professor  A1  Barr  of 
California  Institute  of  Technology,  we  have  develop^  a  procedure  for  measuring  the  locations  of 
the  LEDs  in  the  ceiling.  The  success  of  the  new  ceiling,  currently  under  construction,  relics  on  this 
calibration.  These  locations  must  be  known  to  within  a  millimeter  or  two  for  acceptable  tracking 
performance,  in  terms  of  smooth  and 
accurate  tracking. 

The  calibration  algorithm  begins  with  a  rough  estimate  of  the  beacon  locations  and  a  large 
collection  of  raw  HMD  sensor  data.  A  relaxation  technique  is  used  to  adjust  the  LED  position 
estimates  until  they  are  consistent  with  the  raw  sensor  data.  The  initial  LJED  position  estimates  arc 
existed  to  be  accurate  to  within  1  inch  of  true  value.  The  raw  HMD  sensor  data  arc  collected 
while  the  HMD  is  being  moved  about  beneath  the  cciiing.  No  additional  hardware  is  required,  and 
only  practicable  precautions  need  to  be  taken  while  collecting  the  data. 

The  basic  algorithm  works  as  follows:  After  collecting  the  raw  data  (about  45  minutes  arc  required 
to  collect  HMD  sensor  data  from  25,000  positions),  we  use  a  method  called  space-resection  by 
collinearity  to  compute  where  each  of  the  observations  was  made.  The  LED  positions  arc 
inaccurately  known,  so  the  computed  observation  positions  will  be  inaccurate.  The  collinearity 
algmithm,  however,  has  the  property  of  averaging  out  some  of  the  error.  Now  consider  the 
oteervation  positions  to  .x;  fixed,  and  allow  the  LED  positions  to  vary.  We  ’’fit"  each  of  the  LED 
locations  to  the  fixed  observation  positions.  This  too  has  the  property  of  averaging  out  random 
error.  This  concludes  a  single  iteration  of  the  algorithm.  These  two  steps  are  repeated  until 
adjustments  of  both  LED  positions  and  observation  positions  are  minimal.  This  typically  takes 
about  20  iterations,  which  require  2  hours  of  computing  time  on  an  HP  model  TCiO  for  25,000 
observations  on  a  ceiling  with  960  LEDs.  The  algorithm  is  linear  in  both  observation  count  and 
LED  count. 

In  the  course  of  developing  our  algorithm,  we  discovered  two  si^ificant  sources  of  error  which 
must  be  controlled.  First,  the  algorithm  is  less  effective  when  using  raw  HMD  sensor  data 
obtained  while  the  headmount  was  in  rapid  motion.  Ideally,  the  sensor  data  should  be  taken  when 
the  headmount  is  stationary,  but  this  would  make  data  collection  too  lengthy  a  process:  25,000 
observations  is  a  tj^ical  observation  count  needed,  and  holding  the  headhunt  still  in  25,000 
different  locations  is  not  feasible.  Therefore,  the  data  is  acquit  "on  the  fly,"  that  is,  while  the 
HMD  is  being  move  around  beneath  the  ceiling.  Experience  shows  that  moving  the  headmount 
slowly  produces  better  data  than  moving  it  fast. 
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The  belter  solution  is  to  account  for  movement  in  the  mathematical  model  of  the  hcadmouni 
motion.  The  current  model  assumes  that  the  HMD  does  not  move  significantly  in  the  20- 
millisecond  window  during  which  data  are  taken  for  each  observation.  Experience  shows, 
however,  that  the  movement  is  significant.  More  sophisticated  models  would  help  to  reduce  this 
dynamic  error. 

The  second  source  of  error  is  related  to  the  spatial  distribution  of  the  observations.  Each  LED  must 
be  seen  in  many  separate  observations  and  from  diverse  directions.  If  it  is  poorly  sampled,  then 
the  error  averaging  property  of  the  LEDs'  fit  to  the  observations  is  lost,  and  the  LED  will  not  be 
properly  calibrated.  In  response  to  this  problem  we  use  goaJ-dirccicd  data  collection.  We 
developed  an  X-Windows  application  which  displays  a  top  view  of  the  ceiling  and  highlights  the 
locations  of  poorly  sampled  LEDs,  as  well  as  the  LEDs  currently  visible  to  the  hcad-mount«l 
cameras.  This  runs  while  the  operator  is  collecting  data,  allowing  him/her  to  move  the  HMD  to 
those  undersampled  areas  of  the  ceiling  and,  thereby,  obtaining  a  better  distribution  of 
observations. 

As  a  test  of  the  algorithm’s  robustness,  we  tilted  three  ceiling  panels  in  the  existing  system  by 
inserting  2  1/4-inch  standoffs  under  one  edge.  A  photograph  of  the  ceiling  with  these  tilted  panels 
is  shown  in  Figure  4.  Recall  that  the  panels  arc  2  feet  on  a  side.  The  calibration  procedure  was 
then  conducted  to  determine  the  new  locations  of  these  LEDs.  The  initial  estimated  LED  positions 
were  those  of  the  undisturbed  ceiling  adjusted  by  up  to  1.7  inches  in  a  random  diicction.  It  is 
believed  that  this  is  a  more  strenuous  test  of  the  calibration  algorithm  than  the  new,  larger  ceiling 
will  be.  The  result  of  the  calibration  is  shown  in  a  computer-generated  model  of  the  LED 
locations.  Figure  5.  A  true  measure  of  the  algorithm's  accuracy  is  difficult  to  obtain,  but  the 
algorithm  does  produce  a  measure  of  confidence.  In  the  calibration  of  the  three  tilted  panels, 
described  above,  it  is  believed  that  the  LEDs  positions  arc  known  to  within  about  1  millimeter. 


Figure  4.  The  existing  optical  ceiling  with  three  panels  deliberately  tilted  to  test  our  calibration 
algorithm. 
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Figure  5.  Result  of  calibration:  a  picture  of  the  measured  3D  positions  of  all  the  LEDs  in  the 
ceiling.  The  three  tilted  panels  arc  clearly  evident. 


Alternate  Mathematical  Methods 

Space-resection  by  collinearity  {Azuma9II  has  served  us  well,  but  it  is  computationally- intensive 
and  does  not  always  converge  to  the  corect  solution.  Two  professors  in  the  department.  Dinesh 
Manocha  and  Jack  Goldfeather,  are  interested  in  working  with  us  to  find  direct,  noniterative 
solutions  that  outperform  the  current  method.  Such  methods  may  be  inspired  by  relative  motion 
techniques  used  in  the  computer  vision  field. 

In  order  tc  •"(prove  the  accuracy  of  our  tracking,  we  intend  to  explore  methods  that  take  head 
motion  inti  ..ccount.  One  alternative  is  an  extension  to  the  present  method.  Right  now,  space- 
resection  by  collinearity  computes  a  zero-order  approximation  to  the  HMD's  motion — it  finds  the 
position  and  orientation  most  consistent  with  the  sensor  data.  The  extension  would  provide  a  first- 
order  approximation  to  the  HMD  motion — finding  the  position  and  orientation  and  their  rates  of 
change,  which  is  most  consistent  to  the  sensor  data. 

Another  alternative  makes  use  of  our  ability  to  get  accurate  timestamps  of  when  each  LED  is 
sampled  by  applying  a  Kalman  fiLer  to  the  sequence  of  sensor  data.  This  method  associates  with 
each  moment  in  time  a  state  vecti  which  includes  position,  orientation,  and  their  first  and  second 
derivatives.  Since  Kalman  filters  allow  scalar  measurement  updates,  it  is  possible  to  update  the 
state  after  sampling  a  single  LED,  rather  than  the  dozen  or  s'>  that  we  currently  use  before  each 
update.  Because  sampling  an  LED  takes  about  one  millisec«..nd,  this  method  could  potentially 
reach  update  rates  of  up  to  1000  Hz.  Latency  might  also  be  significantly  reduced.  The  main 
drawback  is  that  this  technique  cannot  double -check  its  computed  answers,  so  it  will  have  to 
occasionally  run  space-resection  by  collinearity  or  some  other  method  to  insure  that  it  does  not  drift 
off  course.  These  alternative  methods  should  also  improve  the  accuracy  of  ceiling  LED  calibrauon. 
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Predictive  Tracking  with  Inertial  Sensors 

Latency  in  viitual  cnviionmcni  systems  is  a  major  problem.  The  effects  arc  more  severe  in  see- 
through  systems  than  in  closed-view  systems.  In  see-through,  we  try  to  register  virtual  objects 
with  the  world.  If  the  virtual  objects  move  with  some  delay  but  the  re^  world  responeb  with 
no  delay,  however,  this  registration  is  ttestroyed.  Even  tiny  amounts  of  latency  can  cause  large 
registration  errors.  At  a  rapid  head  rotation  rate  of  300  degrees  per  second,  keeping  angular 
registration  crrc«rs  below  0.5  degrees  requires  a  system  latency  of  less  than  2  ntillisecwids! 

While  system  latency  can  be  reduced,  it  is  not  currently  practical  to  reduce  latetKy  below  the 
threshold  of  detectability  for  envircximents  of  any  significant  complexity.  Just  updating  a  display 
at  60  Hz  adds  16.67  ms  to  the  system  latency.  The  end-to-end  delay  from  the  time  our  otiling 
tracker  begins  a  measurement  to  the  time  when  the  last  pixel  has  been  drawn  on  both  images  in  tte 
ceiling  HMD  is  about  ms  fOT  a  simple  environment. 

Predictive  tracking  methods  offer  the  possibility  of  compensating  (cx  much  of  this  delay.  Acx:urate 
pn^cdon  requires  a  good  understanthng  of  time  in  the  system.  You  can't  succeed  if  ycHi  don't 
know  how/^  in  the  future  to  predict.  In  an  asynchronous  system  such  as  ours,  this  prediedon 
distance  varies  from  iteradon  to  iteradon.  We  have  synchronized  the  clocks  in  the  ceiling  tim:ker 
with  its  graphics  en^e,  Pixel-Planes  5,  completely  bypassing  UNIX  and  all  of  its  non-ieal-timc 
baggage.  Synchronizadon  is  accurate  to  undt^  one  mUlisecor^  Cbmmunicadon  is  done  via 
shar^  memory.  Because  the  clocks  run  at  slighdy  different  speeds  on  the  two  systems,  we  found 
that  we  must  resynchronizc  every  two  or  three  minutes. 

Accurate  prediedon  requires  good  sensors.  Besides  the  tracker  itself,  we  will  include  velocity  and 
acceleradon  informadon  from  inertial  sensors.  We  have  acquired  rate  gyros  aiai  linear 
accelerometers,  plugged  them  into  a  16-bit  A/D  card  in  an  IBM  PC,  and  verified  that  they  work. 
Mounting  rigs  were  designed  and  built  to  attach  them  to  the  ceiling  HMD  or  any  other  HMD.  We 
arc  cuncntly  building  the  supporting  electronics  to  interface  them  with  the  ceiling  tracker’s  A/D 
equipment 

At  the  moment  we  have  a  simple  prediedon  method  based  on  a  Kalman  filter  that  runs  on  the 
ceiling.  With  the  see-through  camera  activated  in  one  eye,  we  noticed  that  this  prediedon  did  seem 
to  reduce  latency  effects  but  generated  other  artifacts  (such  as  overshoot).  In  the  next  several 
months,  we  intend  to  explore  more  advanced  prediction  methods  that  use  inertial  sensor 
information.  Also,  we  must  look  for  ways  to  evaluate  and  compare  the  effectiveness  of  prediction 
methods,  both  in  objective  terms,  such  as  how  much  it  reduces  the  average  misiegistration,  and  in 
psychological  terms.  The  latter  will  require  user  studies. 

We  are  also  interested  in  using  inertial  senses  to  improve  the  tracking  pt^ormance  of  the  ceiling. 
A  hybrid  optical-inertial  tracker,  as  discussed  earlier,  offers  many  potential  advantages.  While  (xir 
gyros  are  }^bab]y  good  enough  to  suppon  this,  our  accelerometers  and  supportmg  electronics  fall 
fkr  below  the  ^uii^  accuracy  specifications.  The  existing  A/D  converters,  which  are  accurate  to 
less  than  12  bits,  will  not  be  sufficient.  If  using  inertial  sensors  to  suppon  delay  compensation  is 
successful,  then  we  hope  to  make  a  concerted  effon  in  building  or  acquiring  a  mcae  accurate 
inertial  tracker  to  explore  the  potential  of  a  hybrid  system. 

Evaluation  of  Tracker  Delays 

Measurements  have  been  made  of  the  end-to-end  delays  in  head-mounted  display  systems  currently 
in  use  at  UNC-Chapel  Hill.  System  components  that  were  tested  include  the  various  tracking 
devices  currently  in  use:  the  Polhemus  3  Space,  the  Polhemus  FasTrak,  the  Ascension  Bird,  the 
Ascension  Flock  of  Birds,  and  the  UNC  Optical  Ceiling.  These  were  run  on  a  Sun  4  host 
computer  with  our  Pixel-Hancs  5  graphics  engine  and  some  typical  display  devices  (e.g.,  the 
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LCD/CRT  monitors  in  use  in  head-mounted  display  systems).  Significant  findings  include  the 
reduced  delays  in  certain  commercial  tracking  systems  (under  10  ms)  and  the  unandcipatai  impact 
of  the  slow  speed  of  LCD  devices  (adding  as  much  as  a  full  field  time  to  the  system  end-to-end 
delays).  We  are  in  the  process  of  writing  these  results  into  a  technical  report 

Tracker  Distortion  Testing 

We  attempted  to  characterize  and  compare  the  distortion  of  commercial  magnetic  tracking  systems 
due  to  ambient  magnetic  interference.  Tests  consisted  of  mounting  a  magnetic  sensor  on  top  of  a 
movable  tripod  and  moving  it  around  in  the  tracker  workspace.  The  rep«^  position  values  were 
recorded  to  a  file  which  was  then  used  to  generate  tite  position  of  spheres  in  a  virtual  wivironment 
This  in  effect  yielded  a  "smoke  trail"  of  the  reported  tracker  positions.  TTic  smoke  trails  of  the 
various  tracking  systems  were  then  compared  visually  in  a  virtual  envirwimenL  Results  showoi 
that  the  Ascension  Technology  trackers  are  substantitJly  less  susceptible  to  distortion  of  the 
magnetic  field  due  to  the  presence  of  conductors  in  the  workspace  than  the  Polhemus  trackers  are. 

Noise  Sources 

We  collected  and  analyzed  the  raw  data  obtained  from  the  photodiode  sensors  to  identify  any  nense 
sources  that  nmy  be  a^ecting  our  readings.  The  data  was  ^so  analyzed  to  study  the  effects  of  the 
active  gain  ccmtrol  done  by  our  software  to  vary  the  current  used  to  li^t  each  LED  and  the  time- 
variation  of  the  intensity  of  the  light  enuned  by  the  LEDs  over  the  period  of  each  san^ling.  The 
results  of  these  analyses  showed  that  our  data  acquisition  is  relatively  noise  free.  We  did  ronovc 
some  noise  from  our  signals  by  replacing  the  power  supply  for  the  A/D  circuitry. 

Self-Tracker 

Self-Tracker  research  has  been  hindered  by  the  depampe  fiom  our  depamnem  the  graduate 
student  who  was  working  on  the  project  We  are  seeking  a  replacement  but  have  been 
unsuccessful  to  date.  Dr.  Gary  Bishop,  who  originared  Self-Tracker  research  with  his  1984  thesis 
[Bishop84],  hopes  to  guide  a  new  student  in  the  continuation  of  this  research. 


4.4  Interactive  Building  Walkthrough 

The  UNC  Interactive  Buildinc  Walkthrough  sul^roject  aims  at  the  development  of  systems  to  help 
architects  and  their  clients  ex;  ore  a  building  design  prior  to  its  construction  and,  thvefore,  correa 
problems  on  the  computer  instead  of  in  concrete  [Brooks86].  In  previous  versions  of  the  system, 
the  scenes  were  shown  with  ladiosity  illumination,  lights  that  coidd  be  switched  on  or  off  with 
only  100  msec  delays,  textured  surfaces,  and  near  real-time  display  from  changing  viewpoints. 
Up^te  rates  realized  by  the  Pixel-Planes  S  gr^hics  scene  generator  are  30-40  updates/secortd 
(15-20  stereo  images/second)  on  scenes  consisting  of  thousands  of  polygons,  many  of  which  are 
textured.  The  only  downside  to  the  system  was  that  thm  was  a  large  up-front  cost  to  build  and 
hand  tune  riKxkls  and  to  calculate  the  radiosity  illumination  for  the  scene — the  latter  cost  measuml 
in  hours  to  days.  If  the  user  wanted  to  change  the  geometry  of  the  model,  then  the  turn-around 
time  was  usually  measured  in  days  ot  weeks. 

Recently  we  have  moved  toward  creating  a  more  automatic  and  interactive  system  in  which 
changes  in  the  model  geometry  result  in  delays  of  seconds  to  minutes  rather  than  hours  to  days, 
and  the  radiosity  illumination  solution  is  computed  in  near-real  time  while  the  user  is  viewing  the 
nvxlel.  The  end  result  is  a  much  more  practical  and  flexible  design  environment 
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Our  system  uses  a  head-mounted  display  and  the  UNC  ceiling  tracker  to  allow  the  viewer  literally 
to  walk  freely  about  a  10’  x  12'  area  of  a  model.  An  orange  “ribbon”  demarcates  the  boumiarics  of 
the  tracking  volume. 

We  have  long  held  that  virtual-worlds  projects  need  to  move  forward  simultaneously  in  four 
dimensions:  faster,  prettier,  handier,  and  realcr.  In  the  past  year,  we  have  made  progress  in  the 
following  areas: 

•  Faster: 

We  have  worked  cm  speeding  up  the  batch  preparation  processes  rather  than  the  run-time 
calculadcms.  AJthou^  initial  nxxleling  will  s^l  take  place  off-line,  changes  to  the  model,  mesh 
generation,  and  radiosity  arc  all  done  interactively.  In  the  next  six  months  we  are  hoping  to  extract 
partitioning  information  autcmnaticaiiy  from  the  new  version  of  Virtus  WalkThrough. 

•  Prettier: 

We  have  begun  wemking  on  incorporadng  texture  into  interactive  radiosity. 

•  Handier: 

The  big  change  in  this  system  has  been  the  method  of  moving  the  ceiling's  vinual  space  through 
larger  iiKxlels.  The  viewer  translates  the  10’  x  12’  area  in  the  direction  of  gaze  with  the  press  of  a 
button.  Our  research  suggests  that  only  translating  this  area,  rather  than  allowing  translations  and 
rotations,  is  less  disorienting  to  the  viewer. 

We  have  moved  toward  a  more  automatic  system  of  modeling  and  rendering  which  can  be  insened 
without  significant  effort  into  existing  applications  and  can  be  used  to  view  existing  models  not 
specifically  designed  for  the  Walkthrough  systenx 

We  have  signiff  candy  reduced  the  design  iteration  cycle  time  by  migrating  onto  the  graphics 
processors  procedures  that  were  previously  hand  tuned  aiKl  treated  as  batch  processes. 

•  Realer: 

Real-time  mesh  generation  and  interactive  radiosity  have  been  iimorporated  into  the  system  and,  as 
a  result,  models  are  beginning  to  be  dynamic  entities.  The  time  scale  for  the  modeling  pipeline  has 
dropped  from  days  or  weeks  to  seconds  or  minutes.  Changes  in  the  model  arc  no  longer  limited  to 
conectiOTS  of  modeling  errors;  changes  in  design  arc  encouraged  rather  than  prohibit^ 

As  a  product  of  our  collaboration  with  the  Virtus  Coiporation,  we  arc  fast  approaching  a  system 
which  allows  for  interactive  design,  with  a  user  under  the  UNC  ceiling  tracker  making  design 
decisions  and  a  modeler  using  Virtus  WalkThrough  to  car^  them  out.  Soon  the  user  will  be  able 
to  explcae  a  model,  manipulate  objects,  suggest  modificaotHts,  and  have  those  modificadons 
implemented  in  the  model  around  him/her. 


4.5  Virtual-Environment  Ultrasound  Scanning  System 

Our  current  system  woiks  in  the  following  way:  A  video  see-through  HMD  system  is  used  to 
display  ultrasound  data  in  its  real-world  context  with  die  patient  on  the  examining  table.  A  TV 
camera  is  mounted  on  a  ccHiventional  HMD  to  provide  video  see-through  capabiUty.  Syndietic 
images  are  generated  which  correspond  to  the  position  and  wientation  of  the  TV  camera  as  tracked 
by  *e  HMD  system.  The  synthetic  video  images  and  the  live  (TV  camera)  ima^s  are  then 
combined  in  a  single  image.  This  creates  a  virtual  environment  in  which  synthetic  geometry 
appears  fixed  in  the  user's  "real  world"  environment  as  that  user,  wearing  the  HMD,  moves 
around. 
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The  vinual-cnvironment  ulirasound  scanning  system  works  by  gathering  ultrasound  data  and 
transfOTming  it  to  match  the  viewing  position  of  the  TV  camera  mounted  on  the  video  see-through 
HMD.  When  viewed  through  the  HMD,  the  oansformed  ultrasound  data  appears  fixed  in  its 
world,  its  3D  location  within  a  subject.  Two-dimensional  ultrasound  data  is  input  into  our  system 
with  a  frame  grabber  and  a  tracking  system.  Currently  we  use  Polhemus,  Ascension  and  UNC 
Optical  trackers.  The  frame  grabber  transfers  live  2D  ultrasound  video  images  to  our  image- 
generation  system,  Pixel-Planes  5.  The  system  resamples  2D  images  into  a  3D  volume  that  is 
rendered  by  Pixel-Planes  5  and  then  displayed  along  with  live  vidro  in  the  HMD. 

The  ideal  ultrasound  system  would  include  interactive  display  of  volume  data  in  a  virtual 
environment  and  interactive  acquisition  of  3D  ultrasound  volumes.  We  arc  working  along  several 
paths  to  reach  this  goal.  Current  limitations  include  poor  HMD  resolution,  poor  tracking 
technology,  slow  rendering  speeds,  and  the  lack  of  a  3D  ultrasound  machine.  UNC  is  actively 
pursuing  improved  HMD  and  tracking  technologies.  The  ultrasound  group  is  working  on 
automatic  c^bration  of  the  real  world  and  ultrasound  data  with  the  virtual  environment.  We  are 
also  working  on  improving  rendering  speed,  image  quality,  and  the  generation  of  3D  volumes 
from  a  series  of  tracked  ultrasound  images. 

Ultrasound  was  chosen  as  the  acquisition  medium  because  it  provides  real-time  data  acquisitiem 
under  the  physician’s  control.  Ultrasound  scanners  are  compact  and  readily  c  'ailablc. 

Furthermore,  ultrasound  technology  is  not  invasive  and  is  much  safer  than  other  imaging 
technologies  such  as  X-ray.  Our  collaborators  at  Duke  University  and  other  researcl^rs  are 
woridng  on  ultrasound  scanners  that  acquire  volume  data  in  real  time. 

Volume  rendering  was  deemed  necessary  because  the  data  from  ultrasound  is  inherently  noisy. 
When  the  3D  ultrasound  machine  being  developed  at  Duke  University  becontes  available,  3D  data 
can  be  input  direedy  into  our  system  and  the  resampling  of  2D  data  will  no  longer  be  required.  To 
visualize  3D  structures  within  the  body  and  to  allow  the  user  to  view  these  structures  from  arbitrary 
directions,  we  apply  volume  rendering  within  the  head  mounted  display.  The  HMD  gives  the  user 
the  freedom  to  move  around  the  patient,  and  the  volume  data  shows  the  entire  body  structures  and 
not  just  cross-sections  of  them. 


4.6  The  Nanomanipulator 

We  are  collaborating  with  Prof.  Stan  Williams  of  the  UCXA  Department  of  Chemis^  to  build  a 
nanomanipulator,  llic  nanomanipulaior  consists  of  Williams'  ultra-low-drift  scanning  tunneling 
microscope  (STM)  driven  in  real  time  by  our  manipulator  arm,  while  the  viewer  watches  real-time 
reconstruction  of  the  surface,  generated  on  Pixel-Planes  5  and  viewed  either  on  a  large  screen  in 
high  resolution  or  on  the  head-mounted  display.  We  staned  this  activity  in  NovembCT  1991,  got 
the  STM  here  in  Chai^I  Hill  in  January  1992,  and  now  have  viewing,  feeling,  and  arm-driven 
STM  placement  worlang.  The  system  is  also  capable  of  making  controlled  changes  to  a  surface  in 
real  time  through  the  use  of  bias  pulses. 

We  will  attempt  pick-up  and  release  of  small  amounts  of  matter  by  short  pulses  of  voltage  about 
five  times  the  normal  imaging  voltage.  Our  hope  is  to  etch  quantum  circuits  using  this  feature. 
Already  we  have  been  successful  in  placing  material  on  the  surface  with  this  technique.  Williams 
has  been  working  hard  on  this  and  believes  he  can  accomplish  it. 

This  work  is  making  use  of  head-mounted  display  technology  developed  under  this  DARPA 
contract.  Our  facilities  for  haptic  display  with  the  fcMce-feedback  manipulator  were  developed  with 
support  of  NIH.  An  NSF  grant  under  the  Small  Grants  for  Exploratory  Research  (SGER) 
program  has  paid  for  moving  the  STM  here  from  UCLA  and  for  the  salary  of  the  one  graduate 
student  currently  working  on  this  project. 
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5.0  Dissemination  of  Research 

5.1  Publications 

Bajura,  Mike,  Henry  Fuchs,  Ryutarou  Ohbuchi.  "Merging  Virtual  Reality  with  the  Real  World: 
Seeing  Ultrasound  Imagery  within  the  Patient."  Proceedings  of  SIGGRAPH  '92,  (published  as 
Computer  Graphics  26,  no.  2.  July  1992,  203-210). 

Barrett,  H.H.,  J.  Yao,  and  J.P.  Rolland.  "Applications  of  the  Hotelling  Observer  in  Medical 
Imaging.”  Abstract  published  in  Proceedings  of  the  Optical  Society  of  America  General  Meeting, 
New  Mexico,  September  1992. 

Fuchs,  Henry,  Gary  Bishop,  et  al.  "Research  Directions  in  Virtual  Environments:  Report  of  an 
Invitational  Workshop  on  the  Future  of  Virtual  Environments."  Computer  Graphics  26,  no.  3 
August  1992),  153-177.  (This  workshop,  funded  by  an  NSF  grant,  was  held  at  UNC-Chapel 
Hill  23-24  March  1992.) 

Robinett,  Warren,  Russell  Taylor,  Vernon  Chi,  William  V.  Wright,  Frederick  P.  Brooks,  Jr.,  R. 
Stanley  Williams,  Eric  Snyder.  "The  Nanomanipulaton  An  Atomic-Scale  Telcoperator."  Printed 
in  ACM  SI(jGRAPH'92  Course  Notes  for  the  course  "Implementation  of  Immersive  Virtual 
Environments." 

Rolland,  J.P,,  ami  C.  Burbeck,  "Depth  and  Size  Perception  in  Virtual  Environments."  Abstract 
published  in  Proceedings  of  the  Optical  Society  of  America  General  Meeting,  New  Mexico, 
September  1992. 
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5.2  Submitted  for  Publication 

Azuma,  Ronald.  "Head  Tracking  for  Alternate  Realities."  Sutwnitted  for  publication  in  a  special 
issue  of  Communications  of  the  ACM  on  Embodied  Reality  and  Ubiquitous  Computing. 

Gottschalk,  Stefan,  and  John  F.  Hughes  (Brown  University).  "Autocalibration  for  Virtual 
Environments  Tracking  Haidware."  Submitted  to  the  SIGGRAPH^S  conference. 

Mine,  Mark,  and  Gary  Bishop.  "Just-in-Time  Pixels."  Submitted  to  the  SIGGRAPH’93 
conference. 

Yoo,  Tory  S.,  and  T.  Marc  Olano.  "Instant  Hole:  Window  onto  Reality."  Submitted  for 
publication  in  a  special  issue  of  Communications  of  the  ACM  on  Embodied  Reality  and  Ubiquimus 
Computing. 

Taylor,  Russell  M.,  Warren  Robinett,  Vernon  L.  Chi,  Frederick  P.  Brooks,  Jr.,  William  V. 
Wright,  R.  Stanley  Williams,  Erik  J.  Snyder.  "The  Nanomanipulaton  A  Virtual-Reality  Interface 
for  a  Scanning  Tunneling  Microscope."  Submitted  to  the  SICXjRAPH  "93  conference. 


5.3  Presentations 

By:  Mike  Bajura 

Topic:  Mergirig  Virtual  Reality  with  the  Real  World:  Seeing  Ultrasound  Imagery  within 

the  ^tient 

Event:  SIGGRAPH'92  Conference 
Place:  Oticago,  IL 

Dale:  30  July  1992 

By:  Gary  Bishop 

Topic:  Head-mounted  Display  Research  at  UNC-CH 

Event:  IBM  Training  Day  for  Employees  Working  in  Network  Systems 

Place:  IBM,  Research  Triangle  Park,  NC 

Dale:  3  December  1992 

By:  Fred  Brooks 

Topic:  Does  Virtual  Reality  Have  Any  Real  Virtue? 

Event:  Panel  at  1992  Visualization  in  Biomedical  (Computing  conference 
Place:  Chapel  Hill,  NC 

Date:  14  October  1992 

By:  Fred  Brooks 

Topic:  Virtual  Worlds  Research  at  UNC-Chapel  Hill 

Event:  Invited  talk.  Workshop  on  Virtual  ReaJityA^irtual  Environnoents  in  Army  Training 
Place:  Durham,  NC 

Date:  28  October  1992 

By:  Fred  Brooks 

Topic:  Virtual  Worids  Research  at  UNC-Chapel  Hill 

Event:  Invited  talk.  Lectures  in  Computer  Graphics,  Geometry,  and  Visualization  series 
Place:  Princeton  University,  Princeton,  NJ 
Date:  11  November  1992 
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By:  Jeff  Butterworth,  Stefan  Gonschalk,  and  Russell  Taylor 

Topic:  Head-mounted  Display  Research  at  UNC-CH 

Event:  Lecture  Series  on  Virtual  Reality  at  University  of  North  Carolina  at  Greensbwo 
Place:  Greensboro,  NC 

Date:  September  1992 

By:  Henry  Fuchs 

Topic:  Virtual  Reality:  The  Lunatic  Fringe  or  the  Ultimate  Display 

Event:  Invited  talk.  Lectures  in  Computer  Graphics,  Geometry,  and  Visualization  series 

Place:  Princeton,  NJ 

Etaie:  23  November  1992 

By:  Jannick  RoUand 

Topic:  Applications  of  the  Hotelling  Observer  in  Medical  Imaging 
Event'  Invited  talk.  Optical  Society  of  America  general  meeting 
Place:  New  Mexico 

Date:  September  1992 

By:  Jannick  RoUand 

Topic:  Depth  and  Size  Perception  in  Virtual  Environments 
Event  Invited  speaker,  Opti(kl  Society  of  America  general  meeting 

Place:  New  Mexico 

Dale:  September  1992 

By:  Jannick  RoUand 

Tq)ic:  Depth  and  Size  Perception  in  Virtual  Environments 
Event  Visit  to  Philips  Medical 
Place:  Best,  The  Netherlands 
Date:  September  1992 


5.4  A  Partial  List  of  the  Visitors  Who  Observed  Graphics  Demonstrations,  June 

1992-January  1993 

•  Delegation  of  13  Hungarian  participants  in  the  National  Academy  of  Sciences-Hungarian 
Academy  of  Sciences  workshop  on  "Technological  Innovation  and  Growth  in 
Knowledge-Based  Economies" 

•  Kurt  Akeley,  Silicon  Graphics 

•  75-90  participants  in  the  "Virtual  Reality  and  Synthetic  Environments  in  Training" 
conference,  sponsored  by  the  Army  Research  Office  in  Research  Triangle  Park,  NC 

•  Professor  Dr.  sc.  Rainer  Ortleb,  Federal  Minister  of  Education  and  Science  and  Deputy 
Chairman  of  the  Liberal  Democratic  Pt^,  Germany 

•  Hans  R.  Friedrich,  Chief  of  Staff,  Minis^  of  Education  and  Science's  Department  of 
Higher  Education  and  Science,  Germany 

•  -200  attendees  of  the  1992  "Visualization  in  Biomedical  Computing"  conference  that  was 
hosted  by  our  department  in  October  1992 

•  Col.  John  Mentz,  Office  of  the  U.S.  Secretary  of  Defense 

•  David  Meisel  and  Henry  Sowizral,  Boeing 

•  H^on  Lie,  Norwegian  Telecom  Research 

•  20  members  of  NC  State  University's  student  chapter  of  ACM 

•  9  juniors  and  seniors  in  a  multimedia  class,  LeJeune  High  School,  Fayetteville,  NC 

•  Ase  Svensson,  Klas  Odelid,  and  Sverker  Almquist,  Lund  University,  Sweden 

•  Dr.  Brad  Walters,  UNC  Neurosurgery 

•  Mike  McGrath,  National  Science  Foundation 
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•  “15  8th-l2th  graders  from  the  Duke  University  Talent  Identification  Pro^am's  "Visual 
Math"  weekend  session 

•  Donald  Boulton,  UNC  Vice-Chancellor  and  Dean,  Division  of  Student  Affairs,  and  17  of 
his  department  heads 

•  Mark  Mabry,  Lockheed  Sanders 

•  12  from  National  Institute  of  Environment  Health  Sciences,  Research  Triangle  Park,  NC 

•  Nicolas  Ayache,  INRIA,  France 

•  4  from  Motorola 

•  5  from  Virtus  Corp.,  Cai^,  NC 

•  Group  of  12  computer  science  majors  from  Univ.  of  South  Carolina  at  Spartanburg 

•  Barbara  Hermstein  Smith,  Director  of  the  Duke  University  Center  for  Interdisciplinary 
Study  in  Science  and  Cultural  Theory 

•  Alan  Cox,  Honeywell 

•  Chris  Daft,  General  Electric 

•  Group  of  5  faculty  and  students  from  East  Tennessee  State  University  Department  of 
Computer  Science,  Johnson  City,  TN 

•  15  from  Glaxo  Pharmaceuticals 

•  5  from  Information  Spectrum,  Inc.  (designers  of  flight  simulations  for  the  U.S.  Navy) 

•  Pete  Saraceni,  FAA 

•  Prof.  Peter  Foldiak,  Psychology  Department,  Oxford  University,  UK 

•  David  Johnson,  NASA  Langley,  Hampton,  VA 

•  David  Stein,  Director  of  NC  School  of  Math  and  Science,  Durham,  NC 

•  7  from  the  Horida  Community  Colleges  Academic  Technology  Committee 

•  Richard  Bland  and  Cathy  Czeto,  Lighthouse  Low  Vision  Pn^ucts 

•  Mike  Levine,  Henry  Green  and  J.  Pekar,  Ocutech  (optical  company  designing  products 
for  persons  with  low  vision).  Chapel  Hill,  NC 

•  Jim  E>rew,  EHscovery  Place  (science  museum),  Charlotte,  NC 

•  Harry  Yae  and  Mike  Booth,  University  of  Iowa,  Dept,  of  Mechanical  Engineering 

•  13  undergraduates  from  the  Duke  Engineering  Research  Center's  Summer  Institute 

•  16  gifted  and  talented  high  school  students  participating  in  the  Math  and  Science  Ventures 
Program  sponsored  by  UNC-Charlotte 


5.5  Media  Coverage  of  UNC-CH's  Research 

There  is  continuing  public  interest  in  our  department’s  research  accomplishments,  particularly 
virtual  worlds  research.  During  this  reporting  period,  information  on  our  research  has  appeared  in 
a  wide  variety  of  media:  television,  video  shown  at  conferences  and  exhibitions,  newspapers,  and 
magazines. 

Television/Video: 

The  UNC  News  Service  highlighted  the  department's  research  in  virtual  reality  as  one  of  only  three 
areas  of  research  included  in  a  new  video  shown  during  half-time  at  televised  UNC  athletic  events. 

Video  footage  of  UNC's  virtual  worlds  research  was  shown  at  a  10-day  exhibition  called  "Virtual 
Vertigo’’  in  La  Coruna,  Spain,  and  was  subsequently  shown  as  part  of  a  documentary  on  this 
exhibition  for  a  Spanish  television  news  program. 

Video  footage  of  our  research  was  shown  at  the  CNN  World  Economic  Development  Congress 
that  was  held  in  Washington,  DC,  17-20  September  1992. 
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WAGA-TV,  the  CBS  affiUate  in  Atlania.  Georgia,  shot  footage  in  the  graplucs  lab  and  tniorv'icwcd 
Gary  Bishop  fw  a  ihrcc-part  sicry  on  vinuai  reality  that  was  broiKicast  on  their  6:00  news  program 
23-25  November  1992. 

Newspaper/Magazine  Articles: 

"Virtual  Reality:  It's  All  in  the  Mind,"  Ailama  Consntuiion,  29  September  1992. 

"Virtual  Reality:  Exploring  the  Future,"  Furmiure  Today  The  Weekly  Business  Newspaper  of  ihe 
Furniture  Indus^,  5  October  1992.  Focuses  on  architectural  walkthrough. 

"Virtual  Reality:  How  a  Compuier  Gencratco  World  Could  Change  the  Real  World,"  Business 
Week,  5  October  1992.  Includes  a  quote  from  Hen^  Fuchs  on  the  future  of  vinual  reality. 
"Sec-Through  View:  Virtual  Reality  May  Guide  Physicians'  Hands,"  Scientific  American, 
Septemter  1992.  About  the  use  of  head-mounted  display  in  ultrasourKl  exam. 


6.0  Appendixes 
Appendix  A 

Fuchs,  Henry,  Gary  Bishop,  ct  al.  "Research  Dirccuons  in  Virtual  Envuoniwnts:  Repon  of  an 
Invitational  Workshop  on  the  Future  of  Virtual  Environments."  Computer  Graphics  26,  no.  3 
(August  1992),  153-177. 

Appendix  B 

Bajura,  Mike,  Henry  Fuchs,  Ryutarou  Ohbuchi.  "Merging  Virtual  Rcal'fv  >o»n  the  Real  World; 
Seeing  Ultrasound  Imagery  within  the  Patient."  Froceemnjt^  cf  SIGGHAPH  *92.  (published  as 
Computer  Graphics  26,  no.  2.  July  1992,  203-210). 
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Executive  Summary 

At  the  request  of  NSFs  Intmctive  Systems  Program,  a  two-day  invitational  wtnkshop  was  held  March  23-24.  1992  at  UNC 
Chapel  Hill  to  identify  and  recommend  future  research  directions  in  the  area  of  "virtual  environments'  (VE)  *  .  Workshop 
participants  included  some  18  experts  (plus  4  NSF  officials)  from  universities,  industry,  and  other  leading  technical 
organizations.  The  two-day  schedule  alternated  between  sessions  of  the  entire  group  and  sessions  in  the  following  specialty 
areas,  around  which  the  recommendations  came  to  be  organized:  1)  Perception ,  2)  Human-Machine  Software  Interface  . 

3)  Software,  4)  Hardware,  and  S)  Applications.  Also,  two  pardcipanis  develops  a  taxonomy  of  VE  applications  that  is 
included  as  an  tppendix  to  the  rqxxt. 

RecommendatioBs  Suminary: 

Perception: 

Vision 

1.  Collaborative  science-technology  development  programs  should  be  esabiished  at  several  sites  around  the  country  to 

encourage  closer  collaboraticin  between  developers  and  scientists. 

2.  Tlieotetical  research  should  focus  on  devekqrment  of  metrics  of  performance  and  task  demands  in  VE. 

3.  Paradigmatic  applications  and  theoretical  questions  that  iliustraie  the  science-technology  synergy  need  identiScation. 

Audition 
Spatial  Sound 

1  .TlMoretical  research  :tiiould  emphasize  the  role  of  individual  differences  in  Head-Related  Transfer  Functions  (HRTFs), 
critical  cues  for  distance  and  extemalization,  ^lectral  cues  fw  enhancing  elevation  and  disambiguating  the  cone-of- 
conAiskm,  head-motion,  and  interseasory  interaction  and  adaptation  in  the  accurate  perception  of  virtual  acoustic  sources. 
The  noticMi  of  artificially  enhanced  localization  cues  is  also  a  prmnising  area. 

2.  A  fruitful  area  for  jmnt  b^ic  and  applied  research  is  the  develo^ent  of  perceptually-viable  methods  of  simplifying  the 

synthesis  technique  to  maximize  the  efficiency  of  algoritiuns  for  complex  room  modeling. 

3.  Future  effort  should  still  be  devoted  to  developing  more  realistic  models  of  acoustic  environments  with  implementation  on 

more  powerful  hardware  platforms. 

Kfonsoccch  Audio 

1.  Theoretrcal  research  should  focus  on  lower-level  sensory  and  higher-level  cognitive  determinants  of  acoustic  perceptual 

organization,  with  particular  emphasis  on  how  acoustic  parameten  interact  to  determine  the  identifrcation,  segr^ation, 
and  localization  of  multiple,  simultaneous  sources. 

2.  Technobgy  develqmrat  should  focus  on  hardware  and  software  systems  qrecifkally  aimed  at  real-time  gefreration  and 

control  for  acoustic  information  display. 

Haptics 

1 .  envelopment  should  be  encouraged  of  a  variety  of  computer-controlled  mechanical  devices  for  either  basic  scientific 

investigation  of  the  human  haptic  system  or  to  serve  as  haptic  inteiiaces  for  virtual  environments  and  telelt^ieration. 

2.  Research  programs  should  be  initiated  to  encourage  collaboration  among  engineers  who  are  capable  of  building  high 

precision  robotic  devices  mid  scientists  who  can  conduct  txomechanical  and  perceptual  experiments  with  the  devices. 

3.  Research  prt^rams  should  also  be  developed  to  enable  collaboration  among  researchers  working  on  visual,  auditory,  and 

haptic  interfaces,  together  with  computer  specialists  who  can  develop  software  capable  of  synchronized  handling  of  all  the 
sensory  and  motn'  modalities. 

Motion  Sickness  In  Virtual  Environments 

1.  The  virtual  enviitNunent  community  should  be  made  aware  of  the  sensory-motor  adaptation  and  motion  sickness 

probbins  to  be  expected  presently  because  of  hardware  limitations  and  in  the  future  as  belter  virtual  {vesence  in 
nauseogeiuc  environments  is  achieved. 

2.  Research  programs  should  be  initiated  to  evaluate  the  inddence  and  severity  of  sickness  associated  with  different  types  of 

virtual  envirorunents,  and  to  assess  the  kinds  of  sensory-motor  adaptations  and  aftereffects  a.ty>piiiii^  with  virtual 


By  virtual  environments,  we  mean  real-time  interactive  graphics  with  three-dimensioaal  models,  when  combined  with  a 
diqilay  technology  that  gives  the  user  immersion  in  the  model  world  and  diiea  manipulation.  Such  research  has  proceeded 
under  many  labels:  vtrtuo/  reality,  synthetic  experience, .  etc.  We  prefer  virtual  environments  for  accurmy  of  description  and 
truth  in  advertising. 
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environments. 

Evaluation  of  Virtual  Environmcnta 

Research  should  be  conducted  on  the  development  of  psychophysical  techniques  that  measure  the  level  of  el^wt  required  to 
achieve  a  given  level  of  performance,  that  relate  performance  on  simple  tasks  with  perfcKinance  in  a  multi-task  situation,  and 
that  operate  in  a  systematic  and  well-defined  manner  with  complex  stimulus  contexts. 

Human-Computer  Software  Interface: 

1.  Researchers  should  focus  on  the  development  of  new  metaphors  for  VEs  and  the  idaidficaiion  of  reusable,  application- 

independent  interface  components,  specifically  those  whi^  can  be  encqsulaied  in  software  and  distributed. 

2.  NSF  should  support  a  software  ciea^ighouse  for  code  sharing,  reuse,  and  software  capitalizadon. 

3.  We  will  need  to  devek^  metrics  to  guide  the  exploration  of  VE  tools,  techniques,  and  metar^rcrs. 

Software: 

1.  The  development  of  new  modeling  tools  for  model  construction  for  vinual  environments  should  be  supported,  especially 

inside-the-environment  modeling  tools.  These  tools  need  to  be  developed  to  the  point  where  their  effectiveness  can  be 
evaluated.. 

2.  A  facility  for  sharing  existing  and  new  models  should  be  established. 

Hardware: 

Tracking  Systems 

1.  Inertial  tracking  systems  are  prime  for  research  activity  now  becau.se  of  recent  advances  in  micro-acceleronieters  and  gyros. 

Inertial  adjuncts  to  other  tracking  methods  for  sensing  of  motion  derivatives  is  also  a  needed  research  activity. 

2.  Research  into  tracking  technologies  that  allow  large  wenking  volumes  in  outside  spaces  should  be  eitcouiaged. 

Haptic  Systems 

1.  Siqjport  basic  biomechanical  and  psycho-physical  research  on  human  haptic  senses. 

2.  Support  development  of  interactive  force  reflecting  devices,  and  devices  to  disoibute  forces  spatially  and  temporally  within 

each  of  the  (possibly  multiple)  ccmtact  regions. 

Image  Generators 

1.  Research  into  low  latency  tendering  architectures  should  be  encouraged. 

2.  Research  is  neededJnto  software  techniques  for  motion  prediction  to  overcome  inhereitt  system  latencies  and  the  errors 

they  produce  in  registered  see-through  plications. 

Visual  Display  Devices 

NSF  should  pritnatily  support  pilot  projects  that  ofTcr  potential  for  order  of  magnitude  improvement  in  resolution,  Iroghtness 
and  speed.  NSF  should  also  investigate  display  techniques  that  may  ofier  decreases  in  latency  and  to  characterize  problems 
with  (hsplay  phenomena  such  as  frame  sequential  color. 

Applications: 

1.  Applications  are  needed  which  provide  discriminatory  power  to  evaluate  VE  technology  versus  'through  the  window' 

interactive  graphics  and  other  similar  teclmologks. 

2.  Researchm  should  kxA  toward  plications  whidi  solve  real-world  problems.  VE  must  move  beyond  the  stage  of  an 

interesting  te^' -■'^logical  toy  and  begin  to  solve  problems  for  people  where  they  ate. 

3.  Researchers  begin  wtwk  on  the  probable  impact  of  VE  technology  on  society:  Will  VEs  change  the  way  we  work 

(telecom  ^teleconferencing)  or  our  interperscMial  interactions?  As  the  technology  becomes  more  readily  available, 
how  wili  society  react? 

4.  Can  the  use  of  VEs  to  communicate  between  people  approach  the  level  of  communication  we  currently  experience  in 

persraim'inagroup?  What  research  must  be  done  to  move  toward  that  goal?  IsitevenadesirEdriegoal? 
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1.  Introduction 


What  is  Virtual  Envirooments  Research?  In 
1965.  Ivan  Sutherland  in  a  paper.  The  Ultiinate  Display", 
given  at  the  triennial  conference  of  the  International 
Federation  Infonnation  I^essing  Societies,  proclaimed 
a  program  of  research  in  computer  graphics  whi^  has 
chaUenged  and  gukted  the  geld  ever  since.  One  must  kxdt 
at  the  display  screen,  he  said,  as  a  window  through  which 
one  beholds  a  virtuo/ world.  The  challenge  to  computer 
graphics  is  to  make  the  picture  in  the  window  look  real, 
sound  real,  and  the  objects  act  real  Indeed,  in  the  ultimate 
di^lay,  one  will  not  look  at  that  world  thiwgh  a  window, 
but  wU  be  immersed  in  it.  will  change  viewpoint  by 
natural  motions  of  head  and  body.and  will  interact  directiy 
and  natuiaUy  with  the  objects  in  the  world,  hearing  and 
feeling  them,  as  well  as  seeing  them. 

Real'time  interactive  graphics  witii  three-dimensional 
models,  when  combined  with  a  display  technology  that 
gives  the  user  immersion  in  the  model  world  and  direct 
manipulation,  we  call  virtual  environments.  Such  research 
has  proceeded  under  many  labels:  nrtual  reality,  synthetic 
experience,  .etc.  We  prefer  virtual  environments  for 
accuracy  of  description  and  truth  in  advertising.  Menriam- 
Webster's  New  Collegiate  Dictionary,  Ninth  Edition. 
deSnes 

virtual  as  'being  in  effect  but  not  in  actual  fact",  and 

environment  as  "the  conditions,  circumstances,  and 
influences  surrounding  and  affecting  an  organism". 


Why  is  VE  Research  hot  now?  From  1965  until  the 
mid-1980's,  the  limited  power  of  computers  and  of 
graphkal  engines  meant  that  Sutherl^s  vision  could  only 
be  realized  for  crude  dqnctions  (V  for  patnhilly  slow 
interactions  for  many  worlds.  Mai^  graphics  lesearchcR 
worked  on  making  more  faithful  visual  depictions  by 
solving  the  problems  of  perspective,  hiding,  raster-scaniung 
pictures,  sh^ng,  os  illumination.  They  got  fidelity  of 
motion  by  animation  onto  film,  computing  minutes  per 
frame,  giving  up  interaction.  Others  worked  on  real-time 
motions  and  interactions  in  toy  worlds  of  only  a  few 
hundred  eiemems. 

Advances  in  technology,  in  computer  and  graphics 
(Hganization.  in  diqjlays,  and  in  interactive  devices  now 
enable  us  to  do  in  a  vi^  frame  ta^  that  used  to  require 
batch  computing.  Digital  signal  processing  algorithms  and 
hardware  allow  the  realistic  prodiKtiion  of  three-dimensional 
sound  cues,  and  increasingly  compaa  and  high  performance 
mechanical  sensors  and  actuators  promise  realistic 
simulation  of  manual  interactions  with  objects.  So  it  is 
now  possible  to  bring  these  lines  of  research  together  and  to 
approximate  Sutherland's  vision  of  interestingly  complex 


worids  with  laiher  good  pictures,  sounds,  and  forces,  with 
tanializingly  close  to  real-time  performance. 

Though  we  still  have  far  to  go  to  achieve  The  Ultiinate 
Display",  we  have  sufficiently  advanced  towards  the 
that  is  timely  to  consider  real  systems  for  useful 
applications: 

•What  are  the  characteristics  of  the  applications  that 
will  most  benefit  from  such  man-machine 
systems? 

•What  are  the  technical  barriers  tiuu  stand  in  the  way  of 
these  applications? 

•How  can  these  most  profitably  be  addressed?  Howc») 
NSF  (or  DARPA)  and  the  VE  research  community 
make  a  coordinated  push  through  these  barriers? 


II.  Overview 

In  light  of  the  recent  surge  of  interest  in  Virtual 
Environinents  in  science,  inchistry,  and  the  media,  an 
invitational  workshop  was  held  at  the  University  of  North 
Cartdina  at  Giapel  Hill  on  March  23-24, 1992, 
attherequestof  Dr.  John  Hestenes  (Director,  likeractive 
Systems.  National  Science  Foundation).  Ibe  workshop 
was  chaired  by  Drs..  Gaiy  Bishop  and  Henry  Fuchs  wiA  the 
purpose  of  developing  lecomiiieiidations  for  research 
dbrections  in  this  field.  Eighteen  researchers  from  the  US 
and  Canada  spent  two  days  in  large  and  small  groups 
developing  a  consensus  on  die  recommendations  in  this 
report 

The  participants  divided  into  five  working  groups  in  order 
to  focus  on: 

1.  Percepekm  (chaired  by  Steve  EUis), 

2.  Human-Computer  Software  Interface 

(chaired  by  Randy  I^usch), 

3.  Soifoware  (chaired  by  Mark  Green), 

4.  Hardware  (chaired  hg  Mtchael  Modieli)and 

5.  Applications  (chaired  by  Marcus  Brown). 

Also,  two  participants,  Ivan  Sutherland  and  Warren 
Robinett,  developed  a  taxonomy  of  VE  an>i>cations  that  is 
included  as  an  appendix. 

The  recommendations  of  each  of  the  groups  were  reviewed 
and  dianwsed  by  all  of  the  participants. 

This  report  sununarizes  the  results  of  the  workshop.  These 
results  are  orgtBiized  arotaid  the  five  divisions  of  the 
working  grtMQis.  Each  section  fsesems  the  current  status  of 
the  sub-area,  the  perceived  needs,  and  recommendations  for 
future  research  directions. 
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III.  Perception 


Vision 

Because  of  the  pervasive,  dominant  role  of  viaon  in  human 
affairs,  visual  stimuli  are  without  question  the  most 
impevtant  component  in  the  creation  of  the  computer-based 
illusion  that  users  are  in  a  virtual  environment.  There  are 
four  aspects  of  this  key  nde  of  vision:  the  charactexistics  of 
the  visual  image,  the  strucoire  of  the  visual  scene,  the 
visual  consequences  of  manipulative  and  vehicular 
interaction  with  the  scene,  and  the  role  of  visual 
inftmnation  for  spatial  orientation. 

Status 

Visual  image  Modem  visual  psychofriiysics  makes 
intensive  use  of  computer  gnqthics  to  synthesize  high 
resolution  stimuli  for  experimental  manipulation.  Di^lay 
generation  and  digital  filtering  techniques  have  come  to 
play  an  essential  role  in  nudem  laboratories  studying 
human  vision.  The  mathemadcai  and  cooqNitaiional 
techniques  used  to  describe  the  visual  stimuli  that  are 
studied  have  also  become  the  languages  in  which  theories 
about  visual  phenomena  are  phrased  (Watson,  1989) 

Visual  scene  Structure  in  the  visual  image  is  automatically 
identified  by  biological  image  processing  that  s^r^aies 
foreground  fiom  background  and  qxmtaoeously  groups 
regions  together  into  subparts.  Som  aspects  ^  this  image 
segregation  appear  to  be  the  result  of  parallel  processing 
while  other  show  evidence  of  sequential  processing 
(Treisman,  1985).  Oice  segregated,  the  contours  and 
features  crdlected  into  groups  may  be  interpreted  as  objects 
in  the  space  surrounding  die  observer.  The  separated 
patterns  of  contours  and  regions  may  then  be  interpreted  as 
a  surrounding  space. 

Visual  world  The  spatial  inteipretatian  of  visual  images  is 
highly  dependent  iqxm  the  kinematic  characteristics  of  the 
image  motion,  in  particular  those  motions  that  are 
consequences  of  the  observer  himself  (Cutting,  1986).  The 
patterns  of  image  motion  that  are  assodaied  with  obsavers' 
movements  provide  much  of  the  necessary  infonnatioo  for 
guidance  through  a  cluttered  environment  and  have  provided 
the  basis  for  development  of  what  J.  J.  Gibson  described  as 
a  higher-order  psychophysics.  In  this  field,  tesearehere  may 
investigate  the  natural  linkages  estaldished  between 
properties  (rf  image,  or  objea  motirni,  and  complex  normal 
behaviors  such  a  walking  or  object  avoidance. 

Just  as  motion  of  an  dbstrvet  causes  ghibal  dianges  in  the 
paaerr:  of  rdative  motion  in  the  visual  image,  so  to 
manipulatire  iniemction  with  visible  rejects  also  produces 
characteristic  visible  transfonnations  related  to  the  objea’s 


posiuon  and  identity,  (e.g.  Warren,  ei  al,  1991),  which  have 
been  extensively  studied  to  provide  the  bases  for 
psychological  and  physiological  theories  of  manipulative 
interaction. 

Visual  Qrientatinn  Visual  information  is  not  only 
important  for  local  navigation  while  traversing  an 
environment  but  also  for  global  path  planning  and  route 
selection.  These  more  global  ta^  have  been  studied  in 
isolation  during  scientifically  motivated  experiments  (e.g. 
in  Howard,  1982).  But  visual  orientation  is  also  important 
for  more  integrate  tasks  in  which  subjects  use  visuaii  aids 
such  as  maps  to  maintain  their  internal  representation  of  the 
sutTounding  space  and  assist  planning  of  future  activities. 

Needs 

Visual  imaye  Precision  visual  tasks  will  require 
improvements  in  the  image  quality  of  small  display 
systems  that  provide  photopic  luminance  levels  with 
several  arc-minute  pixel  resolution.  Low  level  visual 
perfdnnance  should  be  assessed  with  visual  parameters 
likefy  to  be  provided  by  future  di^iay  systems  which  may 
use  nonstandbud  pixel  layouts,  variable  field  resolution,  and 
field  magnification  to  optimize  allocation  of  compiaer 
gr^ihics  processing.  Higher  resolution  inserts  in  the 
cer^  visual  field  may  be  utilized,  but  gaze  directed  control 
of  these  fields  may  not  be  necessary  if  they  can  be  made 
suffidently  large,  le.  to  about  30  degrees.  Since  the 
presentation  (d' wide  fields  of  view  (>  tiO  degrees 
monocular),  will  likely  involve  some  geometric  image 
distortion,  studies  the  loleiabie  distortion  and 
characreiistics  of  adaptation  will  also  likely  be  retpiired  for 
specific  tasks.  However,  because  the  binocular  overlap 
between  the  left  and  right  eye  images  need  not  be  complete, 
monocular  fields  exceeding  60!  may  only  rarely  be  required. 

Visual  scene  Since  virtual  environment  will  only  be  able 
to  present  somewhat  degraded  low  level  visual  cues  such  as 
contrast  and  stereopsis,  the  capacity  for  viewers  to  segregate 
foreground  Gnom  background  is  likely  to  be  kss  than  that 
with  natural  images  fiom  teal  environments.  Accordingly, 
visual  segregation  with  d^raded  image  quality  and 
dynamics  should  be  studied  and  enhancements  to  oveicome 
difficulties  should  be  developed. 

Visual  conseoiieinces  The  vistol  consequences  of 
envoonmental  inieiactions  generally  involve  intersensory 
integration  and  do  not  quality  as  strictly  visual  issues. 
However  there  are  purely  viaial  consequences  of  motion  in 
a  simulation  which  are  important  for  perceptual  fidelity:  a 
compelling  visual  simulation  will  require  dynamic  as  well 
as  kinetnaiic  modeling  which  currently  is  difficult  to  carry 
Old  at  the  necessary  interactive  rates,  which  ideally  should 
exceed  30  Hz  simulation  ioq}  fieqoency.  Important  work  is 
required  on  the  subjective  and  obj^ve  operator  reactions  to 
approximated  Itineriiatic  and  dynamic  models  of  synthetic 
envitofunems.  How  far  can  a  simulation  deviate  fiom 
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correct  dynamical  modeling  and  still  appear  to  be  realistic? 

Vistial  nrientation  Imperfect  and  slow  dynamics  of  virtual 
environments  can  lead  to  significant  difficulties  for  users  to 
maintain  their  spatial  orientation  within  a  simulated  larger 
environment  Orientation  aids  to  compensate  for  these 
difficulties  should  be  developed  to  allow  developers  to 
simulate  highly  detailed  real  environments  when  such 
detailed  simul^on  is  required.  These  aids  amount  to 
enhancements  for  orienteering  within  a  virtual  environment 
and  should  assist  users  in  switching  between  ego-  and 
exocentric  frames  of  reference  which  will  be  needed  for 
efHcient  interpretation  and  control  of  objects  in  the 
simulated  environment 

Recommendations 

1.  Collaborative  science-techitology  development  programs 

should  to  be  established  at  several  sites  around  the 
country  to  encourage  closer  coUaboradon  between 
developers  and  scientists. 

2.  Theoredcal  research  should  focus  on  development  of 

metrics  of  performance  and  task  demands  in  VE. 

3.  Paradigmadc  applicadons  and  theoredcal  questions  that 

illustrate  the  xience-technology  synergy  need 
kkndfkadcn. 

Comment:  Theinherentinterdisciplinary  nature  of  VE  will 
benefit  from  cutriculum  modifreadons  to  improve 
communicadon  between  petcepoiai  sdendsis  and  intetface 
designers.  Cunently,  th^  researchers  have  significantly 
different  research  apmda  and  goals  which  can  interfere  widi 
coUaboradon.  Interface  designers  are  happy  with  infcraial. 
imperfect  guidance  not  the  reladve  truth  which  the 
scientists  seek. 


Audition 

Status 

Two  general  areas  of  acousdc  research,  spatial  sound  and 
the  R^-time  generadon  of  nonspeech  audio  cues,  are 
critical  for  virtual  environment  reseaich  and  technology 
development  Speech  generation  and  recognidon,  also 
important  features  of  auditory  displays,  will  not  be 
discussed  here. 

Soadal  Sound  The  simuladon  of  spatial  localization  cues 
for  interactive,  virtual  acousdc  displays  has  received  the 
most  attention  in  recent  work.  Perceptual  research  suggests 
that  syndiesis  of  purely  anecboic  signals  can  result  in 
percqiiual  orors,  in  particular,  increases  in  front-back 
reverb,  decreased  elevation  accuracy,  and  failures  of 
extentaUzadtKL  These  en  its  tend  to  be  exacerbated  when 
virtual  sources  are  generated  from  non-petsonaiized  Head- 


Related  Transfer  Functions,  a  common  circumstance  for 
most  virtual  displays.  In  general,  the  synthesis  technique 
involves  the  digital  generadem  of  sumuli  using  Head- 
Related  Transfer  Functions  (HRTFs)  measured  in  the  ear 
canals  of  indivukal  subjects  or  artificial  heads  for  a  Large 
number  of  real  source  (loudspeakers)  locations  (e.g., 
Wighunan  &  Kistler.  1^9;  Wenzel.  1992).  Other 
research  suggesas  that  such  errors  may  be  mitigated  by 
providing  more  complex  acouroc  cues  derived  from 
reverberant  enviratunents  (Begaulu  1991).  Recently,  some 
progress  has  been  made  in  intetacuvely  synthesizing 
complex  acoustic  cues  using  a  real-time  imfriementation  of 
the  image  model  (Foster,  et  al..  1991) 

Nonspeech  Audio  Following  from  Gibson’s  ecological 
approach  to  perception,  one  can  corx%ive  of  the  audible 
world  as  a  collection  of  acoustic  "objects*.  In  addition  to 
spatial  location,  various  acoustic  features  such  as  temporal 
onsets  and  offsets,  timbre,  pitch,  intensity,  and  rhythm,  can 
specify  the  ktentities  of  the  objects  aixi  cemvey  meanii^ 
about  discrete  events  or  ongoing  actions  in  the  world  and 
their  relationships  to  one  another. 

One  can  systematically  manipulate  these  features, 
effectively  creating  an  auditray  symbology  which  operates 
on  a  continuum  from  "literal*  everyday  sounds  to  a 
completely  abstraa  mapping  of  statistical  data  into  sound 
parameters.  Prirtciples  for  design  and  synthesis  can  be 
gleaned  from  the  fields  of  music  (Blatiner,  Sumikawa.  and 
Greenberg.  1989),  psychoacoustics  (Patter^  1982),  and 
higher-level  cognitive  studies  of  the  acoustical  deterniinams 
of  perceptual  organization  (Bregman,  1990,  Buxton,  Gaver. 
and  Bly,  1989).  Recently,  a  few  studies  have  also  been 
concerned  with  methods  for  directly  characterizing  and 
iixxleiing  envimuneniai  sounds  such  as  walking  sounds 
(Li.  Logan,  and  Pastore.  1991).  Other  relevant  research 
includes  physically  or  smictuxally-based  acoustic  models  of 
sound  source  characteristics  such  as  radiation  patterns 
(Morse  and  Ingaid,  1968). 

Needs 

Snatial  Sound  It  seems  dear  that  simple  anechoic 
simulations  of  spatial  cues  will  not  be  suflicieru  to 
minimize  perceptual  errors  and  maximize  perceptual 
"presence”.  Dynamic  modeling  of  compiu  acoustic 
environments  requires  enormous  cranputational  resources 
for  real-time  im|Aementation  in  a  truly  interactive  (head- 
tracked)  display.  Currently  it  is  not  practical  to  render  more 
than  the  fust  one  or  two  reflections  from  a  very  small 
number  of  reflecting  surfaces  in  real-time.  However, 
because  of  the  less  stringent  requirements  of  the  audWy 
modality,  acoustic  digital  signal  processing  is  now 
advanced  enough  to  allow  significam  strides  in  our  basic 
understanding  of  human  sound  localization.  While  fully 
realistic,  interactive  sunulatkms  of  a  conceit  hall  may  not 
yet  be  feasible,  syntheas  techniques  are  suffidemly 
developed  to  allow  an  unprecedented  degree  of  stimulus 
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control  for  the  purposes  of  psychophysical  studies. 

Nnnspeech  Audio  A  few  cue-generation  systems  have  been 
specifically  integrated  for  virtual  environment  applications 
while  some  designers  are  beginning  to  develop  systems 
intended  for  rfam  *soniiicadon”  However,  far  more  effcxi 
should  be  devoted  to  the  development  of  sound-generation 
technology  specifically  aimed  at  inftvmatioo  display. 
Perhaps  more  critical  is  the  need  for  further  research  into 
lower-level  sensory  and  higher-level  cognitive  detenninanis 
of  acoustic  perceptual  organization,  since  these  results  will 
Kive  to  guide  technology  developmenL  Fuilha,  relatively 
little  research  has  been  concerned  with  how  vatious  acoustic 
parameteis  interact  to  determine  die  idendficatian, 
segregadon,  and  localizadon  of  muldple,  simultaneous 
sources.  Understanding  of  such  interaction  effects  will  be 
critkal  in  any  acousdc  dt^iay  developed  for  both  virtual 
environments  and  telepresence. 

Recomnendaaons 

Spaaal  Sound 

1.  Theoretical  research  should  emphasize  the  role  of 

individual  differences  in  HRTFs,  critical  cues  for 
distance  and  extemalizadon,  qiectral  cues  fv  enhancing 
elevation  and  disambiguating  the  cone-of-confiision, 
head-modon.  and  intersenscHy  interacdon  and 
adaptadoo  in  the  accurate  perception  of  virtual  acoustic 
sources  (see  Wenzel.  1992).  lire  notion  of  super- 
auditory  localizadon,  or  artificially  enhanced 
localizadon  cues,  is  also  a  promising  area  (Durlach, 
1991). 

2.  A  fruitful  area  for  joint  basic  and  a^ilied  research  is  the 

development  of  perceptually-viable  methods  of 
simplifying  the  synthesis  technique  with  the  goal  of 
maximizing  the  efficiency  of  algorithms  for  complex 
room  modeling  (increasing  the  number  and  comi^exity 
of  modeled  reflections). 

3.  In  contrast  to  visual  display  technology,  we  are 

currently  much  closer  to  ^veloping  tnily  realistic 
simulathsns  of  auditory  envinMunenls.  Since  the 
research  pay-off  is  likely  to  be  both  high  and  timely, 
future  effort  should  still  be  devoted  to  devek^ing  more 
realistic  models  of  acoustic  environments  with 
implementation  on  more  powerful  hardware  platforms. 
Some  (rf  the  issues  that  need  to  be  addressed  are 
nonuniform  radiators,  diffuse  leflecdons,  scattering 
reflectors,  dif&actkm  and  partial  obscuration  by  walls 
or  other  objects,  spreading  loss  and  high-fipequency 
absorption. 

ISflpspgcdliAuiiiQ 

1.  Theoretical  research  should  focus  on  tower-level  sensory 
and  higher-level  cognitive  determinants  of  acoustic 
perceptual  tvganization,  with  particular  emphasis  on 
bow  acoustic  parameters  interact  to  determine  the 


identificaiitm,  segregation,  and  localization  of  multiple, 
simultaneous  sources. 

2.  Technology  development  should  focus  on  hardware  and 
software  systems  speciflcally  aimed  at  real-time 
generation  and  control  for  acoustic  information  display, 
using  basic  theoretical  knowledge  as  design  guidelines. 


Haptics 

The  human  haptic  system  is  composed  of  subsystems  that 
enable  tactile  and  kinesthetic  senses  as  well  as  motor 
actions.  In  contrast  to  the  purely  sensory  nature  of  vision 
and  audition,  only  the  haptic  system  is  capable  of  direct 
action  cm  real  or  virtual  enviroiunems.  Being  able  to 
touch,  fed,  and  manqNilate  objects  in  the  environment,  in 
itodition  to  seeing  (and/or  bearing)  them,  gives  a  sense  of 
compelling  immersion  in  the  environment  that  is  otherwise 
not  possible.  It  is  quite  likdy  that  much  greater  immersion 
can  be  achieved  by  the  synchnnious  operation  of  even  a 
simple  haptic  interface  with  a  visual  display,  than  by  large 
improvements  in  the  fidelity  of  the  visual  display  alone. 
Consequently,  it 's  important  to  develop  a  wide  variety  of 
haptic  interfaces  to  interaa  with  virtual  enviroranents. 
Examples  of  haptic  interfacet  that  are  being  used  in  virtual 
environment  research  are  joysticks  and  hand/arm 
exoskeletons.  In  general,  they  measae  and  display  users' 
body  part  positions  as  well  as  the  forces  on  them.  The 
biomechanical,  sensorimotor,  and  cognitive  abilities  of  the 
human  haptic  system  determine  the  design  specifications  for 
the  hardwW  and  software  of  h^Hic  intertKss. 

Status 

Compared  to  vision  and  audition,  our  understanding  of 
haptics  is  very  limited.  The  recent  trend  of  conducting 
multidisciplinary  haptic  studies  invtriving  biomechanical, 
psychophysical  and  neurophysicriogical  experiments 
together  with  computatioi^  nnodels  has  contributed  to  rapid 
progress.  Due  to  the  availability  of  powerful  computers 
and  high  precision  mechanical  sensors  and  actuators,  it  is 
now  possible  to  exert  control  over  experimental  variables  as 
never  before. 

Biomechanics  of  Contact  In  any  task  involving  physical 
contact  with  au  object,  be  it  for  exploration  or 
manipulation,  the  mechanics  of  the  contact  interface  plays  a 
fundamental  role.  Both  in  the  tactile  sensory  information 
flow  bom  the  object,  and  in  imposing  desir^  motor  action 
on  the  object,  the  contact  interface  is  stnmgly  influenced  by 
the  sur&ce  and  voluinetric  physical  properties  of  the  skin 
and  subcutaneous  tissues.  Although  some  data  on  the  in 
vivo  biomechanical  properties  and  several  simple 
computational  models  of,  say.  the  primate  fingeipad  are 
available,  they  ate  inadequate  at  present. 
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Sensing  and  Control  of  Interface  Variables:  Thetemi 
intoface  variables  is  meant  to  include  the  kinematic 
variables  (i.e..  the  relative  positions,  orientations,  and 
motions)  ^  various  tody  pans,  toother  with  the  associated 
^  normal  and  shear  fon^  arising  from  contact  with  objects. 

The  kinematic  information  is  conveyed  by  our  kinesthetic 
sense,  whereas  the  contact  forces  are  sens^  by  both  tactile 
and  kinesthetic  sysmns.  In  the  kinesthetic  space, 
psychophysical  phenomena  such  as  anisotropies  in  the 
perception  of  distance  and  orientation,  apparent  curvature  of 
strai^t  lines.  non-EucUdean  distance  measures  between  two 
^  points  etc.,  have  been  reported.  In  tasks  involving  physical 

contact,  si^  as  manipulation  of  objects,  it  is  known  that 
tactile  information  is  crucial  in  controlling  gr^  forces. 
These  control  actions  can  ranj^  from  a  faa  spiivri  reflex  to 
a  relatively  slow,  deliberate  action.  However,  even  simple 
questions  concerning  our  abilities,  (such  as  what  is  our 
^  resolution  in  the  sensing  and  control  of  interface  variables), 

or  the  mechanisms,  (sich  as  how  we  perceive  joint  angles 
or  ccmttct  forces),  ^  not  yet  have  unequivocal  answers. 

Perception  of  Contact  Conditions  and  Object  Pronerties: 
The  contact  conditions  perceived  through  the  tactual  sense 
can  be  broadly  classifi^  as  static  (with  or  without  skin 
^  stretch  due  to  shear  forces),  slipping  and  vibratory.  The 

object  pn^rties  that  are  infen^  include  both  geometric 
(such  as  shape),  and  material  properties  (such  as 
compliance).  Tlie  perception  of  both  contact  conditions  and 
object  properties  is  based  on  intensive,  temporal,  spatial  or 
spatir.^  temporal  stimulus  variations,  and  the  associated 
^  net  :odes.  Recent  psychof^ysictd  and 

nc  'tvsidlogical  investigations  have  provided  some 
ar.  >10  questions  concerning  perception  and  neural 
cooing  of  roughness,  raised  features  on  rigid  objects,  slip, 
microtexture,  shape,  cmnpliance,  etc.  However,  the 
important  connection  between  the  loads  imposed  on  the 
skin  surface  within  the  regions  of  contact  with  objects  and 
1  the  ctniesponding  perception  has  only  begun  to  be 

addressed 

Needs 

In  order  to  enuitciate  the  specifications  for  the  design  of 
haptic  interfaces,  p^cnmance  of  the  human  haptic  system 
'  should  be  characterized  This  includes  the  determination  of 

(a)  the  biomechanical  properties  of  skin  and  subcutaneous 
soft  tissues  that  govern  the  mechanics  of  contact  with  the 
interface;  (b)  the  abilities  of  the  human  haptic  system  and 
the  strategies  used  by  human  subjects  in  pfenning  haptic 
tasks;  and  (c)  evaluation  of  the  effectiveness  of  haptic 
»  interfaces.  A  majOT  barrier  to  progress  from  the 

perspectives  of  biomechanics,  neuroscieace  and 
psychophysics  has  been  the  lack  of  robotic  stimulators 
capabte  of  delivering  stimuli  under  sufficiently  precise 
motion  and  force  contioL 

Biomechanical  Investigations:  The  tight  mechanical 
'  coupling  between  the  human  skin  and  ha^  interfaces 


► 


strongly  influences  the  effectiveness  of  iIm  tntoface. 
Therefore,  the  specifications  for  the  design  of  sensors  and 
actuators  in  the  interface,  as  well  as  the  control  algorithms 
that  drive  the  interface,  require  the  detmnina^  of  uorface 
and  bulk  properties  of,  say.  the  fingerpad  The 
measurement  of  force  distributions  within  the  contact 
regions  with  real  objects  is  needed  to  determine  how  a 
display  should  be  driven  to  simulate  such  contacts  in  virtual 
environments.  In  addition,  computational  models  of  the 
mechanical  behavior  of  soft  tissues  will  aid  in  simulating 
the  dynamics  of  task  performance  for  testing  control 
algorithms,  as  well  as  in  determining  the  required  task- 
specific  force  distributions  fn-  the  displays.  This  requires 
measurement  of  the  in  vivo  sldn  and  subcutaneous  ^t 
tissue  response  to  time-varying  iKvmal  and  tangential  loads. 

Psychophysical  InYCSligations:  Determination  of  the 
human  abilities  (in  terms  of  resolution,  bandwidth,  etc.)  in 
sensing  and  control  of  net  contaa  force  vectors  as  wdl  as 
joint  angles  or  end  point  positions  will  set  the  design 
specificatians  for  hapdc  interface  devices.  The  rather  targe 
body  of  data  available  on  taedie  sensing  of  vibratory 
stimuli  and  the  data  on  qiatial  localizadon  and  lesoltttion, 
together  with  addidonal  psychophysical  experiments  on  the 
perception  of  contact  condidons  and  object  properdes  will 
influence  directly  the  design  of  tactile  displays.  Theoretical 
developments  concerning  the  task-specific  flow  of  sensory 
informadoa  and  control  of  motor  acdon  are  needed  to 
generate  testable  hypotheses  on  our  hapdc  inteiacdons  with 
both  real  and  vinual  environments. 

Hapdc  Interface  Evaluation:  Although  the  hapdc  interfaces 
available  at  present  are  quite  limited  in  their  capabiiides, 
they  need  to  be  evaluated  firom  the  perspeedve  of  human 
perception.  For  example,  a  foice-reflecdng  joystick 
attached  to  the  floor  can  be  called  a  grounded  dis’lay. 
whereas  a  force-reflecting  exo-skeletal  device  attach^  10  the 
user's  forearm  would  be  an  ungrounded  display.  (It  would, 
in  fact,  be  grounded  at  the  forearm).  The  grounding  choice 
affects  whether  or  not  the  user  experiences  throughout 
his/her  entire  body  the  stresses  induced  by  contact  with  a 
virtual  object  The  ctmsequences  of  using  an  ungrounded 
disj^y  to  simulate  contaa  forces  which  really  stem  from 
ground  sources  are  not  known  and  warrant  investigatkm. 
Fuithermore,  the  fidelity  with  which  the  tactual  images 
have  to  be  di^riayed  and  the  motor  acdons  have  to  be 
sensed  by  the  interface  dqiends  on  the  task,  stimulation  of 
other  sensory  modalities,  and  interaction  between  the 
modalities.  Experimenting  with  die  available  haptic 
interfaces,  in  conjunction  with  visual  and  auditory 
interfaces,  helps  to  identify  the  necessary  design 
improvements. 

Recommendations 

1.  Development  of  a  variety  of  crnnputer-controUed 
mechanical  devices  for  either  baric  scwntific 
investigation  of  the  human  haptic  system  or  to  serve  as 
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haptic  interfaces  for  vinual  environments  and 
teleoperation  should  be  encouraged.  The  biomechanical 
and  psychophysical  research  work  detailed  in  the  Needs 
sectian  above  should  be  supponed. 

2.  Research  programs  should  be  initiated  to  encourage 

coUaboradon  among  engineers  who  are  capable  of 
buikUng  high  precisitm  robotic  devices  and  scientists 
who  can  conthia  bkxnechanical  and  perceptual 
experiments  with  the  devices. 

3.  Research  programs  should  also  be  developed  to  enable 

collaboration  among  researchers  working  on  visual, 
auditory,  sid  haptic  interfaces,  together  with  computer 
sprriaitgs  who  can  develop  software  capable  of 
synchronized  handling  of  ^  the  sensory  and  motor 
modalities. 


Motion  Sickness  in  Virtual  Environments 
Status 

Human  movement  and  locomodon  involve  a  dynamic 
sensory-motor  adaptation  to  the  IG  background  force  of 
Earth.  Disruptions  of  this  reiadonship,  which  depends  on 
cotreladons  among  patterns  of  sensory  feedback  &om 
vision,  touch,  somatosensadon,  prcqpriocqrdon.  and  the 
semicircular  canals  and  otolith  organs  of  the  labyrinth  wtir. 
motor  informadon  about  ongoing  movements,  ir  ^ 
variety  of  perceptual  and  motor  errors  and  ofuai  motion 
sickness  as  well.  Ertws  and  sickness  persist  Ui>dl 
adaptation  to  the  new  situation  is  achieved.  Diflerent  types 
of  rearrangements  difler  in  terms  of  how  dilftcult  it  is  to 
adapt  to  them  and  for  some  situations,  adaptation  cannot  be 
achieved.  There  also  ate  great  individual  dififererices  in 
terms  of  susceptibility  to  motion  sickness,  ability  to  adapt, 
rate  of  adaptation  and  retention  of  adaptation. 

Rearrangements  can  be  of  many  types,  including  for 
example,  delays  in  feedback  loops  as  encountered  in  flight 
simulators  or  teleoperatCM’  systems,  various  types  of  visual 
or  auditory  position  displacements,  alterations  of  the 
effective  inertial  ma-Si  of  the  head  or  body  such  as  are 
brought  about  by  wearing  headgear  or  a  space  suit,  changes 
in  the  motor  commands  necessary  to  achieve  movement 
such  as  in  rotating  environments  or  altered  gravitoinertial 
force  fields.  Aitiflcial  transport  of  the  body  in  vehicles- 
cars,  trains,  boats,  aircraft,  spaceships  -  leads  to  complex 
alterations  in  sensmy-motor  control  which  typically  elicit 
perftmnance  orors  and  motkm  sickness  until  adulation  is 
achieved.  Interestingly,  sickness  also  devekgrs  during 
experienced  motion  of  the  body  when  the  pattms  of 
feedback  present  during  voluntary  movements  ate 
appropriate  for  the  actual  rather  than  experkttced  motion. 
This  highlights  the  fact  that  no  general  theory  exists  which 
predicts  which  situations  will  be  disruptive,  nauseogentc,  or 
dillicult  to  adapt  to,  or  which  individuals  will  be  most 


prone  to  these  {xoblems. 

It  can  be  expected,  however,  that  sickness  will  be  a 

problem  both  in  the  early  development  stages  of  vinual 

environments  and  once  they  have  been  perfected.  In  fact,  • 

reports  of  motion  sickness  are  becoming  commonplace.  It 

can  be  fully  expected  that  as  sickness  brought  about  by  low 

resolution  displays,  visual  lags,  and  improper  fc»ce  feedback 

disappears  because  of  technical  improvonenis,  sickness 

will  be  more  frequent  because  of  the  improved  ability  to 

create  virtual  environments  involving  'vehicle'  motions  ^ 

which  ut  itaturaliy  nauseogenic.  * 

Needs 

Motion  sickness  is  going  to  be  a  hi^y  significant 

problem  in  virtual  environments.  Moreover,  it  must  be 

recognized  that  motion  sickness  is  a  complex  syndrome.  H 

Nausea  and  vomiting  represent  only  one  part  of  motion 

sickness  and  often  not  tlw  most  crucial  aspect  of  sickness. 

Other  symptoms  include  headache,  dizziness,  eye  strain, 
lethargy,  and  fatigue.  Often  a  distinction  is  made  between 
'gut*  and  'head'  symptoms. 

Umkr  laboratory  conditions,  motion  sickness  is  relatively  ^ 

easy  to  recognize.  Experimental  studies  of  motitm  sickness 
typically  involve  highly  provocative  test  situations  which 
rarely  last  more  than  an  hour  or  two.  Thus,  sickness  is 
expected  and  the  personnel  carrying  out  the  experiments  are 
highly  skilled  in  recognizing  signs  and  symptoms  of  acute 
motion  sickness.  Often  the  subjects  are  trained  in  ^ 

identifying  die  subjective  concomitants  of  sickness  as  well. 

Motion  sickness  is  more  difllcult  to  recognize  under 

operational  conditions.  Such  conditions  tend  to  be  less 

provocative  and  to  bring  on  initially  more  head  symptoms 

than  gut  ones.  A  sailor  may  not  realize  that  his  experience 

of  drowsiness  is  an  early  sign  of  developing  motion 

sickness.  Highly  motivated  individuals  may  be  able  to  # 

'tough  out*  head  symptoms  in  order  to  complete  a  work 

schedule,  but  ultimately  the  generation  of  work  schedules 

will  be  constrained. 


An  attempt  should  be  made  to  familiarize  investigators  in 

the  virtual  environment  area  with  the  primary  characuaristics  ^ 

of  motion  sickness.  In  addition,  it  would  be  useful  for 

reconis  to  be  kept  concerning  the  incidence  and 

characteristics  of  sickness  encountered  with  different  test 

platforms.  Other  characteristics  of  the  users,  such  as 

motion  sickness  histories,  should  also  be  gathered.  In  all 

likelihood,  research  programs  involving  the  experimental 

evaluation  of  sickness  different  types  of  dynamic  # 

virtual  environments  will  be  necessary  as  vrell  as  programs 

to  study  ways  of  enhancing  the  rate  of  acquiring  adaptation 

and  of  enhancing  retention  of  adaptatioa  Relevant  Nsre, 

too,  will  be  the  assessment  of  the  consequences  of  adapting 

to  virtual  environments  with  regard  to  sensory-motor 

performance  on  return  to  the  mxmaf  environment.  We  ^ 

anticipate  that  functimially  significant  disruptions  of 
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perfonnance  will  be  associated  with  adaptation  to  dynamic 
vinual  environments.  Their  severity,  as  well  as  ways  of 
eliminating  them,  need  to  be  addres^. 

Recommendattons 

1 .  The  virtual  environment  community  should  be  made 

aware  of  the  sensory-motor  adaptation  and  modon 
sickness  problems  to  be  expected  piesendy  because  of 
hardware  limitations  anc  in  the  future  as  better  virtual 
presence  in  nauseogenic  environments  is  achieved. 

2.  Research  programs  should  be  initiated  to  evaluate  the 

incidence  wA  severity  of  sickness  associated  with 
dinierent  types  of  viruial  environments. 

3.  Research  inograms  should  be  developed  to  assess  the 

kinds  of  sensory-motor  adaptations  and  aftereffects 
associated  with  virtual  environments. 


Virtual  Environments  for  Perception  Research 

Status 

Scientists  using  computer-generated  stimuli  and  computer- 
based  signal  processing  have  been  using  the  elements  of 
virtual  er '  ironment  technology  for  years  as  part  of  the 
normal  course  of  research.  Computer  processing,  eq)ecially 
graphics,  is  regularly  used  to  synthesize  sensory  images  and 
present  experimental  stimuli  Though  the  most  obvious 
presentations  have  involved  visual  stimuli  auditory,  haptic, 
and  vestibular  stimuli  are  also  commonly  presented. 
Typically,  these  experimental  applications  ate  not  focused 
on  synthesis  of  multisensory  enviremments.  They  are, 
however,  necessarily  carefully  calibrated  and  often 
automated  for  the  conduct  of  an  experiment  Digital  signal 
processing  is  also  extensively  used  to  filter  and  analyze 
performance  data  (See  EUdnd,  Card,  and  Hochberg,  1989). 

Virtual  environment  (VE)  technology  is  inherently  an 
interdisciplinary  field  thtf  will  be  likely  to  integi^  the 
previous  scientific  wwk  in  percqttion  and  manual  control 
(Ellis,  1991;  Durlach  et  al,  1992).  There  is  consequently  a 
great  potential  benefit  from  coUalxxation  between 
perceptualAnotcr  researchers  and  interface  developers.  VE 
can  p^uce  physically  unrealizable  stimulus  environments 
that  are  uniquely  useful  for  testing  theories  of  petcefKion 
and  manual  control,  but  good/standard  calibration 
techniques  are  needed  to  avoid  unwitting  investigation  of 
display  artifacts. 

Needs 

The  natural  course  of  this  research  will  require  the 
develt^meni  of  appropriate  evaluation  metrics  and 
calibration  procedures  to  be  suggested  for  use  by  VE  system 
deveioi^rs.  Though  anecdotes  from  successful  users 


indicate  significant  benefits  from  the  technology,  existing 
VE  systems  need  to  be  generally  more  analytically 
evaluated  for  specific  utility. 

VE  development  forces  evaluation  of  assumptions/better 
design  of  tests  and  can  force  psycnological  and 
physiological  theories  to  be  mxe  {Hccisely  stated  and 
integrated  and  can  enable  the  asking  of  totally  new 
experimental  questions. 


Evaluation  of  Virtual  Environments 
Status  and  Needs 

VE  systems  can  be  meaningfully  evaluated  only  by 
determining  their  effects  on  the  human  user.  No  other 
measure  matters.  Rigorous,  reliable  and  inierpretable 
measures  of  these  effects  require  careful  and  exact  contrt^  of 
the  stimulus  and  use  of  well-designed  psychophysical 
procedures.  Although  there  is  a  strong  well-established 
tradition  of  sensory  research  that  conforms  to  these 
requirements,  evaluation  of  VE  systems  presents  demands 
that  cannot  all  be  met  by  existing  techniques.  New 
procedures  must  be  devdoped. 

Many  ttf  the  well-develqped  experimental  techniques  focus 
on  die  detectitm  of  small  differences  in  the  stimulus,  e.g.  in 
contrast  detection,  q»tial  discriminations,  motion  detxtion 
and  discrimination  and  depth  discriminations.  Among  the 
psyclK^hysical  methods  cunently  in  use,  those  that  use 
hi^  contrast  stimuli  are  most  likely  to  be  generally  useful. 
Motion  detection  and  velocity  discrimination,  for  example, 
may  prove  to  be  useful  indiemors  of  the  perceptual  stability 
of  the  system.  Similarly,  size  disciiinination  may  prove  to 
bea  useful  indicator  of  the  quality  of  the  depth  peicqM. 

Also  of  interest  is  the  veridicality  of  the  simulated  percept 
Subjective  judgments  of  absolute  size  or  depth  may  be 
useful,  but  rating  judgments  are  highly  context-dependent 
They  may  miss  substantial  differences  between  real  and 
virtual  environments  because  each  is  judged  only  relative  to 
itself.  See-through  head-mounted  di^lays  that  permit 
combination  of  vinual  and  real  environments  offer  a  means 
of  determiiung  the  veridicality  of  the  simulation  by  direct 
comparison  to  the  real  environment 

However  useful  these  lechnkpies,  none  addresses  dhectly  the 
question  of  how  demanding  the  system  is  to  use. 
Psychophysical  experiments  ate  marked  by  a  strong 
restriction  of  task  and  stimuli.  Vinual  oivironments.  on 
the  other  hand,  typically  present  complex  stimuli  to 
observers  who  ni^  complex  judgments.  Breaking  these 
complexities  down  into  simple  components  that  are 
ame^ie  to  traditional  psychophysical  evaluation  cannot 
answer  many  of  the  most  impeu^t  questitms  about  the 
value  of  the  virtual  environment  because  performance 
depends  on  how  information  is  acquired  in  the  face  of  the 
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complexity. 

On  the  other  hand,  use  of  complex  stimuli  can  make 
interpretsaion  of  tlw  data  difficult  The  observo-  may  use 
stimulus  cues  other  than  the  one  being  manipulated 
intentionally.  This  problem  can  be  avoided  by 
manipulating  only  a  single  localized  feature  and  letting  the 
rest  of  the  stimulus  serve  as  context  In  that  case,  alarge 
number  of  different  contexts  or  environmoits  must  be  used 
toimevent  the  observer  from  teaming  to  ignore  the  context 
Use  of  multiple  varied  contexts  inevitaUy  adds  noise  to  the 
measured  response,  requiring  more  trials,  which  requires 
more  contexts,  etc.  Soon  the  number  of  images  to  be 
stored  becomes  large  enough  to  be  {nobtemai^. 
Restricting  the  contexts  to  a  class  that  is  ^Tfxopriate  to  a 
single  application  reduces  the  variability  at  the  cost  of 
generality.  Techniques  need  to  be  developed  for 
systematically  generating  and  describing  contexts  to  be  used 
in  well  contndted  studies.  A  generally  accessibte  lilvary  of 
contexts  might  add  tnder  to  this  pursuit. 

At  least  as  important  as  stimulus  complexity  is  the 
problem  of  task  complexity.  The  accuracy  with  which  an 
observer  can  perform  a  single  task  tells  us  little  about  the 
resources  required  to  support  that  performance.  For  virtual 
environments  to  be  useful,  they  cannot  place  unreasonable 
demands  on  the  user.  If  it  is  difficult  or  taxii^  for  the 
observer  to  collect  each  desired  piece  of  infmmatioo,  the 
system  may  not  enhance  performance  of  complex  tasks, 
llie  infmmation  [nesented  in  the  VE  must  not  only  be 
available,  it  most  be  easily  available. 

Most  psychophysical  mediods  do  not  deal  with  this  a^iect 
of  the  p^lem.  Some  that  do  exploit  the  temporal 
dimension.  Reaction  times  may  be  useful  indicators  of  the 
difficulty  of  a  task.  This  measure  has  the  notable  advantage 
that  the  task  being  studied  need  not  be  limited  to  detecting 
barely  discriminable  stimulus  differences.  The  major 
disadvantage  to  diis  technique  is  the  need  to  handle  ^leed- 
accuracy  tradeoffs.  Another  approach  is  to  limit  the 
exposure  time  of  the  stimulus.  This  avoids  the  problem  of 
sp^-accuracy  tradeoffs,  and  can  result  in  a  more  systematic 
representation  of  the  speed  of  processing.  The  major 
weakness  of  the  approach  is  that  for  all  but  contrast 
detection  tasks,  a  masking  stimulus  must  be  presented  at 
termination  of  the  stimulus  to  stop  further  processing. 
Choice  cS  an  effective  mask  is  often  difficult  and  because 
one  cannot  know  whether  a  given  mask  is  completely 
effective  fat  a  given  stimulus,  comparison  of  effects  across 
stimuli  is  never  wholly  ctmvincing.  Improvements  in 
these  techniques  andAv  creation  of  new  techniques  that 
make  use  trf  the  temporal  dimension  to  assess  task 
diffkulty  would  be  he^ful. 

More  generally,  we  need  to  develop  ways  to  assess  behavior 
when  the  obsor^  is  performing  meue  complex  tasks,  e.g., 
when  he  is  performing  several  simple  tasks  simultaneously 
or  in  rapid  alternation.  Some  techniques  that  have  been 


used  in  other  areas  of  experimental  psychology,  e.g.,  in 
memory  research,  may  be  usefully  ai^ted  hoc. 

Recommendation 

Research  should  be  conducted  on  the  development  of 
psychophysical  techniques  that  measure  the  tevel  of  effort 
required  to  achieve  a  given  level  (rf  performance,  that  relate 
performance  on  simple  tasks  with  perfonruince  in  a  multi¬ 
task  situation,  and  that  operate  in  a  systematte  and  well- 
defined  maimer  with  comptex  stimulus  contexts. 


rv.  Human-Computer  Software  Interface 

Status 

The  interaction  techniques  for  virtual  environmeius  have  yet 
to  be  extensively  explored.  There  are  several  design 
principles  which  are  likely  candidates  for  structuring  VE 
interf^  metaphors,  such  as: 

•natural  behavior  as  an  interaction  priiKiple 
•rapid  prototyping  for  design  ^pace  exploration 
•knowledge-based  agents  for  interactive  guidance 
•respect  for  the  physiological  context  of  the  i^ysical 
body  in  VE  design 

•supernormal  capabilities  as  a  metaphor  for  sensory 
remapping 

These  principles  are  suggestive,  but  need  to  be  evaluated  in 
task  specific  application  domains. 

Current  difficulties  in  the  design  and  construction  of  virtual 
worlds  include; 

•ineffective  software,  particularly  for  prograiiuning 
dynamics  and  interactivity; 

•complex  hardware  configurations  and  e^iecially 
hardware  latency; 

•world  modeling  and  maintenance  (version  ccmtrol  on 
worlds  is  complex); 

•no  theory  of  world  construction; 

•pi^sical  literalism  (assuming  die  virtual  world  is  like 
the  physicai); 

•naive  models  of  interaction  in  a  virtual  envirorunent; 
and 

•failure  of  intuition  (physical  solutions  are  not  viroial 
solutions). 

Cuneni  research  challenges  include: 

•developing  software  operating  systems  which  facilitate 
effective,  iiKiusive,  real-time  interaction,  multiple 
participant,  multiple  sensory,  high  bandwidth,  low 
lai«K^  VEs; 

•determining  which  aspects  of  the  2D  WIMP  uid 
desktop  mecqihors  are  generalizabte  to  higher 
dimensiona]  virtual  worlds; 


11 


I 


Research  in  Virtual  Environments 


•identifying  the  core  of  generic  virtual  world  design 
tools  and  application  independent  interaction 
techniques: 

•integration  of  multidisciplinary  teams  to  address 
)  human,  software,  and  hardware  interactions. 

including  design  and  analysis  of  experinumts;  and 
•bringing  multidisciplinary  knowledge  to  the 
consmiction  of  VEs,  including  work  from 
database,  data  fusion,  networking,  human  factors, 
computer  supported  collaborative  work,  and 
^  artiftcial  inteUigentx  communities. 

Needs 

The  most  {aessing  needs  ate: 

•software  frx'  design  of  and  interaction  with  virtual 
^  worlds  that  is  modular,  flexible,  and  abstract, 

particularly  interpreted  languages  for  rapid 
prototyping: 

•software  operating  systems  and  infrastructure  to 
support  world  design,  construction,  and 
interaction,  particularly  sctftware  which  reduces 
latency: 

^  •metaphors  which  guide  the  exploration  and 

prototyping  of  particular  tools  and  techniques  for 
use  in  VEs; 

•measurement  techniques  and  theories  for  identifying 
differential  effects  of  world  designs  on  the  sense  of 
presence; 

.  •measurement  techniqi^  for  identifying  resource 

'  expenditure,  cognitive  load,  transfer  of  learning, 

adaptation  effects,  and  '<ther  performance 
parameters  of  different  ccmfiguration  of  VEs;  and 
•task-specific  evaluxuion  of  software  tools. 

Secondary,  less  general,  needs  include  the  development  of: 

•navigation  techniques  for  high  dimensional  data  and 
di^lays; 

•location  and  movement  techniques  and  software  tools; 
•manipulation  techniques  and  software  tools; 

•event  history,  filtering,  and  recording  tools; 
•behavioral  and  dynamic  models  for  determining  the 
^  dispositions  of  objects; 

•specification  languages  for  world  dynamics; 

•editing  tods  fru'  objects  and  for  environmental  spaces, 
including  models  of  inter-objea  communication, 
process  management,  and  composition  rules; 

•a  d^gn  theory  of  sensory  presentation  modes  (which 
^  sensory  suites  are  best  for  conveying  which 

tasks?); 

•languages  and  grammars  for  describing  worid 
inteiacucms; 

•the  virtual  body,  mapping  tools  for  ctmnecting  sensors 
on  the  physical  body  to  an  accurate  physiological 
model  in  software,  and  in  turn,  to  the  virtual 
I  object  being  used  as  a  body  in  the  VE;  and 


•ux>l$  that  can  be  used  for  interaction  and  consmiction 
both  inside  the  VE  and  outside  on  a  monitor 
viewing  the  VE. 

Recommendaaons 

Since  VE  design  and  interaction  is  in  its  infancy,  these 

recommendations  are  focused  on  generic  rather  than  ^xcific 

goal* 

1.  Isolating  and  evaluatmg  applicaticHi-independent 

interaction  techniques  and  metaphors.  Researchers 
should  focus  on  die  development  of  new  metaphors  for 
VEs  and  the  identification  of  reusable,  t^lication- 
independent  interface  components,  specifically  those 
which  can  be  encapsulated  in  software  and  di^buted. 
One  specific  area  with  high  potential  is  tf^  use  of 
voice  input  as  a  parallel  input  modality. 

While  some  of  this  evaluation  wdl  be  extensive 
research  centered  on  the  human's  capabilities,  some  of 
it  will  be  rapid,  less  fcrmal  evaluation  to  help  interface 
designers  choose  between  conflicting  alternatives.  In 
one  sense,  these  different  objectives  underscore  the 
differences  between  basic  science  and  engineering.  We 
explicitly  suggest  that  NSF  recognize  the  contributitms 
by  evaluations  made  at  various  levels  of 
certainty.  Fred  Erodes  refers  to  these  as  cerrouuy 
shells,  including  findings,  observations,  and  ruies-of- 
thumb.  This  research  needs  to  integrate  the  diverse 
skills  and  styles  of  multidisciplinary  teams. 

2.  Software  capitalization  —  NSF  should  support  a 

software  clearinghouse  for  code  sharing,  reuse,  and 
software  capitalization.  The  cost  of  having  each  VE 
laboratory  develop  its  own  infrastructure  is  prohibitive 
to  the  effective  conduct  of  research.  Weencourage 
support  of  world  building  and  maintenance  tools,  to 
en^k  verskm  control,  composiuon  of  components 
developed  by  different  design  groups  (tool  portability), 
ease  of  customization  and  configui^ility,  and 
ex|Hessability. 

3.  Measurement  techniques  to  determine  the  quality  of  VEs 

We  will  need  to  develop  metrics  to  guide  the 
explexation  of  VE  tool^  techniques,  and  metaphors. 
The  quality  of  a  VE  is  likely  to  be  related  to  specific 
tasks,  physiological  comfort,  cognitive  and  perceptual 
load,  resource  expenditure,  imuitiveness  of  look  and 
interactions,  ease  of  reconfiguration,  transfer  and 
adaptation  effects,  and  even  individual  differences  and 
preferences.  In  this  regard,  our  final,  concrete 
recommendatton  is  that  NSF  help  forge  inter- 
disciftiinary  collaborations,  especially  between 
computer  sciemi^  and  psychologists. 


% 


12 


Research  m  Virtual  Env'urmnrtcnu 


V.  Software 

Status 

The  c£»i  effective  devclopmem  of  virtual  cnvuonnwms  *iU 
require  a  new  genemion  of  software  uwls.  These  tools 
must  »ipf»rt  the  ^lecial  hardware  devices  chat  are  used  in 
virtual  envirwiments  wd  the  design  of  the  envtroranems 
themselves. 

A  number  of  software  tools  have  been  developed  to  sqiport 
the  production  of  virtual  envaonmonu.  These  tools  can  be 
divided  into  two  broad  groups  based  on  whether  they  are 
commercial  pcothicts  or  the  result  of  a  research  project. 

Most  of  the  commercial  products  suiqnrt  a  particular 
hardware  configuration  that  is  sold  bv  the  software  vendor. 
They  provide  a  basic  level  of  software  support  for  the 
vendor’s  hardware  and  allow  for  the  deveiopmecK  of  sunpk 
virtual  environments,  htost  of  these  tools  provide  basic 
support  for  die  hardware  devices  and  limited  support  for  the 
development  of  the  virtual  environments  themselves. 

The  research  software  toob  tend  u>  be  more  gerund,  since 
there  b  no  mandate  to  support  a  psticular  hardware 
platform.  These  toob  support  a  range  of  hardware 
configurations,  which  bcilitates  the  sharing  of  reaearch 
results  and  the  resulting  program  code.  Again,  most  uf 
these  toob  only  address  the  device  support  issues,  but  in  a 
hardware  independent  mtumer.  Some  (oob  are  beginaing  m 
be  developed  for  supporting  the  production  of  virtual 
environments,  in  addidon  to  supporting  the  basic  hardware. 
One  of  the  major  advantages  of  this  group  of  roftware  toob 
b  that  they  allow  ruw  researchers  to  quickly  enter  the  field 
in  a  cost  effective  maimer,  since  they  are  ftwly  distributod 
and  will  likely  support  whatever  hardware  configuratian  the 
researcher  has  available. 

Needs 

Modeling.  Modeling  b  the  key  software  issue  at  the 
present  time.  The  modeling  problem  b  not  unique  to 
virtual  environnunb,  but  it  b  crucial  to  the  success  of 
many  VE  applications.  In  many  VE  applications,  there  b  a 
tight  connection  between  the  user  and  the  underiyii^  model. 
The  user  must  be  able  to  move  dirough  the  model,  interact 
with  the  objecb  in  the  model,  and  change  the  model 
interactively.  The  development  (tf  ^xxl  modeling  toob 
will  facilitate  the  development  of  new  applications  and  the 
development  of  new  techniques. 

In  this  report,  modeling  refers  to  the  data  structures  that  are 
used  to  iec(»d  the  geometrical  information  for  the 
environmeitt.  Thb  informatkm  includes  the  shape  of  the 
objects  in  the  environment,  their  moving  parts  and  physical 
properties,  and  the  behaviors  that  they  cm  perform  (how 
they  interact  with  other  objects  in  the  environment  and 
with  the  user).  Thb  information  is  not  only  used  to 


produce  the  visual  pres^taoan  of  Uk  envuwtmcni.  bus  ii  is 
also  in  soitfid  ptoductMMi  aruj  lacuk  feedbatk. 

In  a  VE  application  there  ma)  he  multiple  usm  lor  the 
same  apphcaoon.  The  usm  ctHdd  have  ddTercni  needs  or 
be  performing  diffcreiu  tasks,  which  would  lesutt  in  a 
different  view  of  the  nsodci  for  och  user  In  addition,  each 
user  may  have  personal  preferences  ihtu  diaaie  a  diflertru 
view  of  the  mc^l.  The  modelmg  software  meet  be  abk  to 
support  the  differciu  views  of  dse  users 

There  arc  also  applcrNuons  where  mulupk  users  may  be  in 
the  same  environmcfit  at  the  saioe  ome  This  imptes  that 
the  model  must  be  shared  among  die  users.  Each  usa  mu» 
have  a  comtHimi  view  of  the  ippticauoo  (which  may 
depend  upon  how  the  model  is  viewed).  There  must  be 
roethodi  for  keepmg  these  modeb  iqMCHlate  and  at  the 
same  oriK  allow  the  users  to  modify  aidivKluai  copies  of 
the  modei. 

The  model  will  be  used  to  dnve  several  diffenni  media  such 
re  paphars  and  sound.  The  model  must  contain  the 
mfarmauon  requtrrd  by  all  of  these  media,  plus  the 
mfommxM  required  to  tynchronue  their  preseruauon  This 
u  not  as  sifflfrie  as  evems  m  two  media  occturmi  at  the 
same  tsne.  For  eaampte.  in  ihecasc  of  lighoung.  the 
vbual  rypearance  of  the  lighsung  must  occur  before  lu 
sound  is  produced.  Thb  problem  also  occurs  m  mulo- 
media  systons. 

Objeca  in  a  VE  may  correspond  to  obyecu  that  occur  in  the 
real  world.  When  t^  happens,  we  expect  these  objeos  to 
behave  m  the  same  way  as  real  world  objects  behave.  In 
other  words,  they  imm  follow  the  tradiisonai  laws  of 
physka.  wiwn  an  object  b  dropped,  it  should  fall  tauil  u 
reaches  the  ground.  It  shouldn't  be  possible  for  objects  to 
pass  through  each  other.  In  otder  to  get  these  effects  we 
need  to  be  able  to  model  at  least  part  of  physxa.  This 
includes  the  familiar  laws  of  motion  and  the  abilny  to 
detect  coUiskms  between  objects.  These  techniques  must 
perform  in  real'tune.  so  diey  can  be  mcorporaud  mio  the 
frerfliack  provided  by  the  enwotunenL 

Some  of  the  objects  in  the  VE  must  be  bbie  to  respond  to 
the  user's  actions  and  the  actioiu  of  other  (ktjecu.  There 
should  be  ways  of  spenfying  this  behavior  assigning 
these  behavion  to  the  objects  in  the  model.  Some  of  the 
objects  in  the  enviFonmcmt  may  also  have  autottomous 
behaviors,  such  as  walking,  llib  will  involve  i  aactioits 
between  the  objects  and  simul«ed  time  within  the 
enviranmem.  There  b  some  oreriap  between  the 
teclsixpies  requtied  here  and  the  tecitmques  that  are  beu^ 
devetoped  in  computer  animation  sid  simulaiion. 

One  of  the  issues  in  modeling  for  VE  b  wh«her  the 
modeling  should  be  done  inside  or  outside  of  the 
environment.  There  are  cieatr  advam^es  to  both 
approaches.  Modeling  in  die  environment  gives  the 


13 


I 


I 


i 


i 


» 


i 


Research  in  Virtual  Environments 


designer  a  good  view  of  what  the  user  will  see  when  he  or 
she  is  in  the  environment.  Modeling  outside  the 
environment  allows  the  designer  to  use  multiple  views  of 
th^  environmen  uid  draw  iqxxi  the  modeling  techniques 
tku  have  been  a;:veloped  over  the  past  few  decades. 

Current  inside-tbe-environment  modeling  systems  are  not  as 
good  as  the  ouiside-the-environment  ones.  There  are  two 
main  reasons  for  this.  First,  the  I/O  devices  used  in  V£  are 
not  a.^  well  developed  as  die  devices  used  in  traditional 
interactive  computer  grqihics.  For  example,  head-mounted 
di^lays  have  a  much  lower  resolution  than  CRT  monitors, 
and  most  3D  tracking  technok^es  have  higher  noise  levels 
than  2D  devices.  Seoxid.  the  design  technicpies  used  in  the 
outside-die-environment  modelets  have  been  developed  over 
the  past  few  decades,  while  there  has  been  very  little 
r'.jcrience  with  inskie-the-enviionment  models  There 
should  be  an  efftat  to  develop  design  techniques  for  inside- 
the-environment  modelen. 

One  of  the  main  problems  with  large  models  is  navigating 
in  them.  In  2D  applications,  overview  mtq»  of  the  model 
can  be  i»resented  in  separate  screen  wiiulows.  but  similar 
techniques  haven't  bem  devek^ied  for  3D  environments.  It 
has  bera  suggested  that  the  environment  can  be  scaled  in 
ortter  to  give  an  overview  of  it  This  will  only  work  if 
detail  can  be  culled  from  the  model  when  it  is  scaled.  In 
order  to  do  this  effectively,  the  model  must  be  constructed 
in  such  a  way  that  it  is  easy  to  identify  the  details  and  the 
important  components  of  the  model  The  model  should 
provide  assistance  to  the  navigad^  process. 

Modeling  is  currently  a  time-consuming  process.  We  need 
better  tools  to  siqiport  this  task,  partkulariy  modeling  tools 
for  VE  applications.  Reasonable  modeling  tools  have  been 
developed  for  other  domains,  most  notably  computer-aided 
design  and  product  design.  The  objects  in  both  of  these 
domains  are  static,  and  thus  these  tools  don't  address  the 
complete  range  of  VE  modeling  requirements.  An  effort 
shotdd  be  made  to  develop  better  modeling  tools  that 
address  the  problems  of  VE  modeling.  The  development  of 
these  tools  will  assist  in  other  doni.,uis  that  make  heavy 
use  of  modeling. 

Since  the  production  of  good  models  currently  requires  a 
considerabie  amotmt  of  effort,  there  should  be  some 
mechanism  for  sharing  the  existing  models.  There  are  twe 
aspects  to  this  sharing.  First  is  the  determination  of  a 
standard  fonraa  for  encoding  and  transmitting  the  models. 
There  are  mimerous  formats  that  have  been  developed  for 
the  interchange  of  graphical  information.  One  of  these 
formats  may  be  suitable  for  this  purpoK,  or  a  new  format 
may  need  to  be  ttevdoped.  The  format  that  is  used  should 
be  abte  to  encode  all  the  modeling  information,  including 
hreraichical  stnicunc,  j^iysical  properties  of  the  objects, 
sound  and  behavior.  S«:ond,thm  should  be  one  or  more 
sites  that  store,  maintain  mid  distribute  these  models. 

These  sites  would  be  responsible  for  edieedng  the  existing 


models  and  new  models  as  they  are  developed,  cataloguing 
the  set  of  models  and  distributing  them  to  researchers  on 
request  All  of  this  should  be  performed  over  the  existing 
computer  networks  to  guarantee  the  widest  distribution  of 
the  models. 

There  is  a  need  for  certain  real-time  functionality  in  VE 
systems.  In  particular,  there  is  a  need  for  low  latency  in  the 
processing  of  requests,  the  alnlity  to  assign  priorities  to 
processes,  and  the  management  of  time.  Re^-ome 
operating  systems  exist  that  meet  these  requironents,  but 
teseaicbers  haven't  been  wiUing  to  use  them,  since  they 
would  need  to  give  up  the  fKilities  that  are  provided  by 
existing  main-line  operating  systems,  such  as  UNDC.  As  a 
result,  we  would  like  to  see  th^  real-time  facilities  added 
to  UNDC. 

Our  recommeialatioo  is  not  to  directly  fund  this  work  under 
the  VE  inidadve  but  to  encourage  other  groups  to  fund  this 
effort  under  larger  projects  that  me  also  be  interested  in  this 
technology.  In  particular,  the  high  performance  computing 
initiative  should  be  encouraged  to  fund  this  work.  Other 
areas  that  would  benefit  from  this  work  include  robotics  and 
real-time  simulation. 

RecommaidaaoHS 

1.  The  development  of  new  modeling  tools  that  meet  the 

requiiemetus  of  model  construction  for  virtual 
environments  should  be  supported. 

2.  A  facility  for  sharing  existing  and  new  model*  should  be 

esablished.  This  will  involve  both  the  development  of 
standards  for  model  interchange,  and  the  establishment 
of  one  or  more  sites  where  the  models  will  be 
maintained. 

3.  Support  the  development  of  inside-the -environment 

modeling  tools.  These  tools  need  to  be  developed  to 
the  point  where  their  effectiveness  can  be  evaluated. 


VL  Hardware 

Hanlware  for  vntual  environmeias  is  an  area  of  rapid 
progress,  for  several  reasons.  Much  of  the  technology  has 
other  applicatioas  outside  of  VE,  in  areas  such  as  robotics, 
entenainmeat  and  instnimentadoo.  Areas  such  as  high 
definition  display  devices  are  being  driven  by  the  general 
advancement  of  aernkxinductar  design  and  fabrication. 

Nevenfaeless,  a  substantial  number  of  "show  stoppers"  for 
VE  can  be  found  in  hardware.  For  instanre,  without  some 
breakthroughs  in  the  bio-mechanical  ei^neering  of  haptic 
(force  and  totidi  feedback)  devices,  many  of  the  envisioned 
applications  of  VE  will  remain  science  fiction. 
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We  will  explore  the  following  hardware  categories:  tracking 
systems,  haptic  systems,  image  generatCHS,  visual  display 
devices,  audio  systems  and  ^teech  input  systems. 

Tracking  Systems; 

This  section  concerns  techniques  for  determining  the 
location  and  orientation  of  the  user’s  head,  hands  and 
ultimately  of  any  real-world  object  of  interest  to  the  V£ 
user. 

Status. 

Five  categories  of  devices  exist: 

1.  Magnetic  systems,  typified  by  the  Polhemus  and 

Ascension  ttackers,  use  magnetic  fields  emitted  by 
small  antennas  and  detected  by  one  or  more  receiving 
antennas.  They  are  unreliable  in  the  presence  of  metal 
objects.  Today's  systems  have  a  range  of  less  than  Im 
from  the  emitter  muldple-emitter  solutions  are 
possible. 

Latencies  (time  from  initiation  of  a  motion  until  the 
new  data  arrives  at  the  host  computer  via  a  parallel 
link)  in  cunent  systems  are  on  the  order  of  50  msec. 
Ruinored  improvements  may  bring  this  down  to  5 
msec  within  a  year.  Current  systems  can  achieve 
resolutions  of  tqiproximately  1mm  arxl  0,03  degrees, 
and  accuracies  of  approximately  3mm  and  0.1 
degrees.A  wireless  version  of  a  magnetic  tracker  would 
be  technically  difficulL 

2.  Acoustic  systems,  typified  by  the  Logitech  Mouse,  use 

ultrasonic  pulses.  Range  and  latency  are  qiproximately 
the  same  as  today’s  magnetic  devices,  without 
susceptibility  to  magnetic  interference.  They  are 
susceptible  to  line-of-sight  occlusion,  e.  g.  by  the  arm 
of  the  user.  Wireless  versions  are  feasible.  Ultimate 
latency  is  limited  by  the  speed  of  sound  (about  0  J 
m/msec).  Accuracy  is  limited  by  variations  of  the  speed 
of  sound  with  changes  in  air  density. 

3.  Inertial  systems  have  formerly  been  bulky,  expensive, 

and  had  too  much  drift  Advances  in  micromachines 
show  promise  of  producing  small,  inexpensive 
accelerometers  and  rate  gyros  that  have  sufficient 
sensitivity  to  allow  tracking  via  dead  reckoning  for 
sh(»t  periods.  Wireless  operation  would  be  feasible. 
These  devices  are  ncM  yet  commerciaUy  available. 

4.  Among  mechanical  systems,  the  BOOM  viewer  from 

Fake  Space  Labs  uses  arigid  firamewtxk  both  to 
support  the  viewing  device  and  measme  position  and 
orioitation,  with  a  rqported  accuracy  of  4mm  and  0.) 
degree  resolution  at  each  joint.  Eletrause,  unlike  the 
magnetic  and  acoustical  trices,  no  averaging  is 
required,  the  latency  is  very  small  -  under  1  msec. 
Low-cost,  non-load-bearing  mechanical  trackers  are  also 


available. 

5.  Optical  systems  such  as  the  Optotrack  3000  and  the 
^Ispot  II  determine  position  of  targets  via 
triangulation  techniques  from  cameras  at  known 
locations.  They  det^ine  orientation  by  observing 
muidpie  targets.  Without  large  separ^ons  between 
targets,  fuie  measuretnem  of  orientatitm  is  very 
difficulL  These  systems  provide  only  a  smaU  working 
volume  (1  meter  cube)  ai^  are  suscqxible  to  line  of 
sight  occlusion. 

Optical  systems  such  as  the  experimental  tracker  at 
UNC  mount  the  cameras  on  the  head  and  observe 
targets  at  fixed  locations  on  the  ceiling.  This  system 
determines  both  position  and  orientatian  directly  and 
may  be  extended  to  cover  arbitrary  areas  but  must 
maintain  line  of  sight  and  isn't  dimdy  suitable  for 
tracking  the  hands. 

Needs. 

To  support  most  goals  currently  under  consideration,  a 
position  sensor  would  need  to  have  these  characteristics: 
■wireless  operation 

•ability  to  track  multiple  users  in  same  space  without 
iraerference 

•range  of  up  to  lOm  x  10m  x  Sm  with  reference  to  a 
base  unit,  perhaps  the  sire  of  a  briefcase; 

•no  wide-area  antenna,  ceiling  or  sensor  field  required 
•latency  under  S  msec 
•resolutian  of  1  nun  and  0.01  degree 
•accuracy  of  1  mm  and  .01  degree  in  registered  see- 
through  iqiplicadons,  1  cm  and  0.1  degree  in  non- 
registoed  appUcations. 

•sami^ing  rate  >«  60  Hz 

•direct  sensing  of  state  AND  derivatives 

The  private  secuv  is  making  progress  UTwaids  better 
tracing  systems  for  VE  though  the  technology  hasn't 
changed  significaiuly  for  more  than  10  years. 

Recommendations. 

1.  Inertial  tracking  systems  are  prime  for  research  activity 

now  because  of  recent  advances  in  nuctD-accelerometeis 
and  gyros. 

2.  Inertial  adjuncts  to  other  tracking  methods  for  sensing  of 

motion  d^vadves  is  also  a  needed  research  activity. 

3.  Research  into  tracking  technologies  that  allow  large 

working  volumes  in  outside  spaces  should  be 
encouraged. 


15 


Research  in  Virtual  Envirotunents 


» 


I 


% 


I 


i 


I 


Haptic  Systems: 

This  section  concents  devices  which  provide  the  illusion  of 
manual  exploration  or  manipulation  of  objects  in  virtual 
worlds.  These  devices  employ  human  tactile,  kinesthetic 
and  motor  systems  for  interaction  with  the  virtual 
environment.  They  perform  some  or  all  of  two  basic 
functions: 

1.  measurement  of  the  positions  and  forces  (and  time 

derivatives)  of  the  user's  hand  andAv  other  body  parts  to 
be  used  as  control  inputs  to  VE.  and 

2.  displayof  forces  and  positions  and/or  their  ^tial  and 

temporal  distributions  to  the  user. 

Status.  Fbor  areas  of  work  are  of  interest 

1.  Hand  position/joint  angle  measurement  Instrumented 

wands,  joystkks,  gloves  and  exoskeietons  that  measure 
hand  positkm/joint  angles  are  available  in  the  market 
The  major  prOblans  are  the  intrusion  the  user  feels 
while  wearing,  say,  an  exoskeleton,  and  the  ever¬ 
present  need  for  improvements  in  resolutions  and 
ranges. 

2.  Application  of  forces  and  torques:  Exoskelaons  attempt 

to  both  measure  hand/arm  joint  angles  and  to  load  th^ 
joints  with  feedback  torques  around  the  joinl  by 
applying  forces  at  the  contaa  regions.  With  inputs 
£nm  research  into  the  teleoperatioa  oi  robots,  some 
progress  has  been  made  on  the  exosfceletal  pn^lem, 
but  the  systems  ate  heavy,  expensive,  clumsy  and 
unreliable.  The  computational  problem  of 
compensating  for  the  mass  of  the  control  arm  is 
substantial. 

Simulator  motion  bases  represent  the  most  mature 
component  of  this  sub-area.  The  art  of  providing 
sustained  forces  to  the  subject's  whole  body  through  a 
pilot's  seat  is  well  understood.  The  princip^ 
limitations  ate  that  the  average  acceleration  provided  by 
a  terrestrial  motion  base  must  always  be  at  least  Ig, 
and  that  sustained  accelerations  gte^  than  Ig  ate 
achievable  only  in  centrifuges,  which  induce  other 
severe  problems  such  as  corioUs  effects. 

3.  Tactile  Displays.  These  are  two  dimensional  fields  of 

force  applied  to  the  sldti,  to  simulate  touching.  Both 
normal  ixid  shear  forces  are  necessary  to  simu^ 
general  sensations  of  touch.  The  simulation  of 
difTerent  coefiicients  of  friction  (for  ^ous  materials 
and  skin  conditions:  wet,  dry,  oily)  is  problematic.  A 
few  standalone  demonstrations  of  tactile  displays  have 
been  buUt 

4.  Other  stimulus  distributions.  These  include  thermal, 

several  kinds  of  pain,  and  possible  direct  electrical 


stimuli  applied  as  two  dimensional  fields  to  the  skin. 
Pain  is  the  report  of  a  physiok>gical  limit's  being 
reached  or  exceeded.  "Deliberate  synaesthesra"  would 
use  a  small  vibrating  stimulator  or  small  shock  (rather 
than  pain)  to  report  that  a  "virtual  limb"  ckt 
telec^tation  actuator  device  has  reached  its  limit 

Needs. 

Mechanical  stability  is  important.  Force  feedback  syaems 
need  to  have  vibration  rigmously  controlled,  probably  to 
less  than  a  few  microns  amplitude,  to  prevent  false  cues. 
Forces  on  the  order  of  10  Newtons  are  needed,  with  10  bit 
resolution.  With  respect  to  force  distributions,  a  spatial 
density  exceeding  1  mmAaxel  and  a  temporal  resolution 
approaching  1  kHz  ate  needed. 

Recommendations. 

1.  Supptxt  basic  biomechanical  and  psycho-physical 

research  on  human  haptic  senses. 

2.  Support  devel(q>ment  of  interactive  force  teflectir^ 

devices,  and  devices  to  distribute  forces  spatially  attd 
temporally  within  each  of  the  (possibly  multiple) 
contact  regions. 

Image  Generators 

Status. 

The  computer  graphics  industry  is  nqridly  improving  its 
polygon,  lighting  models  and  texture  rendering  curability, 
and  the  cost  of  visual  systems  continues  to  dn^  at  a  rate 
>50%lyt  for  constant  performance..  The  principal  deficit  for 
VE  applications  concerns  latency  -  the  delay  between  a 
change  to  the  visual  database  or  viewing  parameters  and  a 
change  in  the  display. 

The  latency  of  fast  workstations  such  as  Silicon  Grrqphics 
products  is  one  frame  interval  (33ms  for  30Hz  ipdate  rate) 
to  compute  the  image  in  the  offscreen  buffer,  plus  one 
refresh  interval  (I4tns  at  72Hz  refresh).  The  drive  in  the 
private  sector  towards  higher  through-put  will  not 
automatically  solve  the  latency  problem,  as  pipelined 
architectures  wUl  probably  continue  to  be  used.  TheVE 
conununity  needs  lower  latency  even  at  the  expense  of 
polygon  through-puL 

Needs. 

Most  graphics  users  do  not  need  low  latency  (at  the  levels 
required  by  VE).  and  so  most  gta{4iics  architectures  are 
de^ly  pipelined.  Research  in  comouter  architecture  is 
nee^  to  explore  ways  to  reduce  i^rency. 

There  is  no  good  data  on  the  threidiold  for  perception  of 
latency;  research  is  needed.  Simulator  sickness  studies  on 
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the  effect  of  latency  have  generally  explored  the  domain 
between  100  and  300  msec,  because  of  limitations  in  the 
imaging  systems  in  use. 

One  way  to  detennine  this  threshold  would  be  with  the  use 
of  a  panoramic  buffer.  A  pie-aored  scene  would  be 
"panned”  in  response  to  h^  tracking  informaticMi.  The 
inherent  latency  of  an  NTSC  di^lay  device  is  around  16 
ms;  in  order  to  beat  that  rate,  "racing  the  beam"  would  be 
necessary  (L  e.  the  generation  scan  lines  immediately 
befene  their  rendering,  based  on  most  current  geometry). 

The  lowest  latency  VE  systems  likely  to  be  built  in  the 
near  future  will  stUl  have  delays  from  user  motion  to  visual 
output  that  cause  significant  emvs  in  registered  see-through 
systems  and  perceptual/sickness  efrects  in  all  VE  systems. 

Recommendations. 

1.  Research  into  low  latency  rendering  architectures  should 

beencoutagecL 

2.  Research  into  software  techniques  for  motion  prediction 

to  overcome  inherent  system  latencies  and  the  errors 
they  produce  in  registered  see-through  applications  is 
nee^ 

Visual  Display  Devices 
Status. 

Commercially  available  LCD  displ^  are  presently  in  the 
range  of  200  x  300  pixels  (consisting  of  a  color  triad). 
DARPA  is  fiuiding  deve!q;nnent  of  a  true  640  x  512 
electroluminescent,  and  alw  a  color  LCD  system  consisting 
of  color  quads  (12^  x  1024  in  monochrome).  The  displays 
are  to  be  ready  within  2  years. 

DARPA  is  also  funding  the  development  of  a  reflective 
system  using  very  small  deftamabie  mintxs;  around  10**6 
moving  minors.  Ihe  device  will  have  high  speed  (10 
mkrosec)  and  pown’ capabilities  because  all  energy  is 
reflected,  not  absorbed.  Can  be  used  with  color  by  triplex  or 
field  seq  color. 

Optics  fm*  head  mounted  displays  are  available,  bur  are 
heavy  and  expensive.  Because  the  images  in  VE  are  totally 
synthetic,  it  is  possible  to  perfeam  some  transformations 
(such  as  correction  of  some  kinds  of  chromatic  aberration) 
in  software  which  formerly  required  tqxical  methods. 

Needs. 

1024  X  1024,  color,  60  Hz  or  fastn*.  in  1’  square 
packaging:  lightweight,  large  visual  field  qaics. 


Recommendation 

1.  NSF  should  primarily  suppon  pilot  (aojects  that  offer 

potential  for  order  of  magnitude  improvemem  in 
resolution,  brightness  and  speed. 

2.  NSF  should  also  investigate  ditqrlay  techniques  that  may 

offer  decreases  in  latency  and  to  characterize  problons 
with  display  phenomena  such  as  flame  sequential  color. 


Audio  Systems: 

Status 

Spatial  Sound  Briefly,  the  apiaoach  is  to  synthesize 
externalized,  three-dimensional  sound  OKS  over  heachihanes 
using  very  high-speed  digital  signal  processing  devices  (see 
WenzeL  1992).  In  general,  the  synthesis  technique 
involves  the  digital  generation  of  stimuli  using  Ifead- 
Related  Transfer  Functions  (HRTFs)  measured  in  the  ear 
canals  of  individual  subjects  or  artificial  heads  for  a  large 
number  of  real  source  (loudspeakers)  locations  (e.g., 
Wightman  &.  Kistkr,  1989).  In  most  current  systems 
(e.g.,  the  Convtrivotron),  firom  one  to  four  moving  or 
static  sources  can  be  simulated  (with  varying  d^rees  of 
fidelity)  in  a  head-stable,  anechoic  enviroiunent  by 
convolution  of  arbitrary  signals  with  HRTF-based  Alters 
choemi  according  to  the  output  of  a  head-tracking  device. 
Motion  trajectories  and  static  locations  at  greater 
resolutiaas  than  the  empirical  data  are  generally  simulated 
by  linear  interpolation  between  the  measured  impulse 
responses.  Ato,  in  some  systems,  a  simple  distance  cue 
can  be  provided  via  real-time  scaling  of  amplitude. 

Nonspeech  Audio  The  ideal  synthesis  device  would  be  able 
to  flexibly  generate  the  entire  continuum  of  mmspeech 
sounds  described  earlier  as  well  as  be  able  to  continuously 
modulate  various  acoustic  parameters  associated  with  these 
sounds  in  real-time.  Current  devices  available  for 
generating  nonqteech  sounds  are  based  almost  exclusively 
on  MIDI  (Musi^  Instrument  Digital  Interface)  technology 
and  tend  to  fall  into  two  general  categories;  "samplers", 
which  digitally  store  sounds  for  later  real-time  playback, 
doi  "synthesizers*,  which  rely  on  analog  or  digital  sound 
generation  techniques  originally  developed  for  imitating 
musical  instruments.  With  samplers,  many  different 
sounds  can  be  reproduced  (neariy)  exactly,  but  substantial 
effort  and  storage  media  are  requh^  fir  accuraiely  pre¬ 
recording  sounds  and  there  is  usually  limited  real-time 
control  of  acoustic  pai  'meters.  Synthesizers  are  more 
flexiMe  in  the  type  of  real-time  control  available  but  less 
general  in  terms  tff  the  variety  of  soemd  qualities  that  cm 
be  generated.  A  potential  disadvanta^  of  both  is  that  they 
are  not  specifically  designed  for  the  generation  and  control 
of  sounds  for  information  display  and  tend  to  require  that 
the  iKer  have  qrecialized  knot^ledge  of  musical^production 
techniques. 
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Needs 

Any  generally  useful  audio  hardware  sys^  must  sample 
its  inputs  with  at  least  16  bits  of  resolution  and  40  to  SO 
khz  sampling  rate.  To  perform  the  spatial  syr- thesis 
functitHis  described  in  the  Status  section,  a  co  uputadonai 
rate  of  about  300  MIPS  is  required.  Additional 
computatitHial  power  will  probably  be  needed  to  implement 
iiK»e  complex  environmental  models  as  well  as  the  real¬ 
time  generation  and  control  of  nonspeech  audio  cues. 

Recommendations: 

This  seems  to  be  an  area  where  the  technology  has  arrived 
at  a  price^jerfonnance  level  which  can  suppon  rgpid 
progress.  Research  should  now  be  funded  primarily  to 
improve  models  of  peicqrtion  arxi  techniques  for  generating 
acoustical  models  of  the  environment  and  of  environmental 
sounds  (acoustical  ’objects*),  rather  than  for  extensive 
hardware  development  Ifowever.  while  tx)t  the  primary 
emi^tasis,  some  new  hardware  development  will  probably 
be  required  to  accomnmdate  more  elabtxaiB  and  efBciertt 
models  of  complex  envitonmetus  as  they  devek^. 


Speech  Input: 

Status 

Inexpensive  devices  are  now  available  which,  with 
reasonable  reliability,  can  recognize  individual  words  from 
the  breaker  who  traitied  the  system.  Speaker-independent 
methods  ate  achieving  some  accqxance. 

Needs: 

Many  selection  tasks  would  be  better  mediated  by  voice 
than  by  pointing,  given  the  unsatislacttny  state  of  visual 
pointing  in  3D.  As  the  technology  of  spaxh  recognition 
improves,  textual  input  in  some  situations  could  move 
from  keyboard  to  voice,  which  will  impaa  VE-based 
database  activities  and  other  information  accessing  tasks. 

Recommendadons: 

Support  would  be  appropriate  for  projects  which  exploe  the 
integration  ttf  voice  input  into  the  VE  user  interface. 
Hardware  development  is  probably  not  needed. 


Vn.  Applications 

Status 

Fot  VE  systems  and  technologies  with  respea  to  particular 
applications,  there  are  two  basic  questions: 


1.  Can  the  given  application  be  accomplished  with  VE 

technology?  Many  applications  are  currently  beyond 
the  state  of  the  art 

2.  If  the  application  can  be  accomplished,  will  it  prove 

superior  to  other  technologies  for  accomplishing  the 
same  task? 

While  the  promise  of  VE  technology  has  been  widely 
aclmowledged,  there  are  very  few  production-qualiQr 
applications  that  are  used  regularly  to  solve  r^-world 
problems.  Many  desired  appUcadons  have  tequiremoits 
which  are  currently  beyond  state  of  the  art.  those  that 

are  feasible,  it  remains  to  be  shown  that  VE  provides  a 
superior  solution. 

Compared  to  cmnputer  models  displayed  via  conventional 
screen-bound  interactive  Cthrough-the-window')  graphics. 
VEofiera 

•total  immersion  in  the  computer  model, 

•kinesthetic,  intuitive  viewpoint  change,  and 
•direct  manipulation  in  3-D. 

Thus,  one  would  expect  to  be  able  to  show  comparative 
supeiiraity  in  simple  applications  that  exploit  these 
advantages. 

These  comparative  advantages  promise  an  Older  of 
n«gnitiiA»  better  illusion  for  the  user,  thus  greater 
involvement  of  the  user  with  the  task.  This  promise  alone 
has  been  the  driving  force  for  devek^tment  of  VE 
technology  in  the  past  With  the  exception  of  vehicle 
simulatois  and  entertainment,  most  current  applicatiofis  of 
VE  have  been  developed  for  driving  and  testing  the 
technology  rathm’  than  to  accomplish  ends  related  to  the 
applications. 

The  following  application  areas  staixl  to  gain  significantly 
frxxn  developments  in  VE  technology: 

1.  Data  or  Model  Visualization: 

Scientific  visualization  techniques  ate  widely  accepted 
as  a  means  for  extracting  undostanding  from  la^ge  data 
spaces.  When  VE  technologies  are  coupled  with 
existing  visualization  systems,  the  user  will  experience 
a  funttoental  paradigm  shift  in  interacting  with  the 
data  or  model.  Stereoscofnc  imaging  combined  with 
intuitive  control  over  point  of  view  frees  the  user  to 
concentrate  on  the  data  rather  than  the  computer 
interfioe.  The  incorporation  of  odter  senses  into  the 
’display"  potentially  offers  a  mechanism  for  conelation 
of  data  features  through  non-visual  means. 

Augmentadoo  is  an  extension  data  visualization; 
The  image  presented  to  the  user  is  a  combination  of  a 
computer-generated  image  mtd  the  view  of  the  real 
worid  around  the  user. 
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2.  Designing  and  Planning: 

A  primary  characteristic  of  design  activities  is  the 
iterative  atialyxe-refine  cycle  which  takes  place  as  the 
model  evolves.  Any  method  which  aids  the  designer 
during  this  process  will  improve  the  entire  activity. 
Because  a  strength  of  VE  is  the  capability  for  direct 
manipulation  of  (Ejects  within  the  virtual  space, 
design  activities  sh^d  benefit  greatly  from  this 
teclmology.  Existing  rqrplicatioos  for  architectural 
design  and  surgicai  planning  attest  to  the  benefits  of 
VE  technology. 

3.  Education  and  Training: 

Computer  models  of  teal-wotid  systems  can  be 
presented  through  a  virtual  environment  Given  that 
the  qrprapriafB  level  of  fideli^  can  be  provided,  the 
user  can  interact  with  the  virtual  system  as  though  it 
were  the  real  thing.  In  this  context  the  educational 
implications  of  this  technology  ate  immediately 
obvious.  In  addition  to  simply  pmtraying  the  real 
worid,  the  rales  governing  a  vit^  w^d  can  be  altered 
to  allow  the  user  to  experience  and  leam  from  alternate 
"realities".  In  this  way,  the  effect  and  importance  of 
various  physical  pararneters  can  be  studied,  and  the 
educatitmal  focus  can  be  on  cognitive  skills  rather  than 
the  specifics  of  a  particular  environmeiu. 

4.  Teleoperations: 

VE  can  be  apidied  to  situations  where  an  environment 
is  too  hazardous  for  a  human,  buta  tele*sensor/operator 
may  enter.  As  an  example,  the  expknation  of  the 
surtiaen  Titanic  was  accomplished  ptimaiily  through 
leleoperations.  Although  the  imerface  was  not  called 
VE,'  this  is  an  obvious  extension.  (Although  it  is  an 
interesting  problem;  research  into  the  robotic 
teleopetators  is  beyond  the  scope  of  VE  research.  VE 
research  should  focus  on  the  in^ace  between  die 
human  and  the  computer  model,  not  on  the  model- 
world  interface.) 

5.  Psydiological  Test  Beds: 

VEs  can  produce  physically  unrealizable  stimulus 
envinminents  duu  are  uniquely  useful  for  testing 
theories  of  human  perceixion  and  manual  control.  This 
includes  research  into  visual,  auditory,  and  htgaic 
sensory  stimuli,  and  its  effects,  both  short-term  and 
long-term,  on  the  user  of  a  VE. 

6.  Entertainment  and  Artistic  applications: 

Given  the  rnnount  of  public  mid  media  interest  in  VE 
technology,  there  is  an  eumomic  potential  for 
entertainment  uriiich  bUJs  itself  as  "virtual  reality". 
There  are  existing  applications  of  this  type  currently 


installed  in  majcv  theme-parks.  One  can  only  expect  to 
see  this  market  expand  in  the  fiuure  as  public  exposure 
grows. 

7.  Artistic  Applications 

Artists  are  often  the  first  individuals  to  explore  new 
media  and  technologies.  The  basic  activi^  of  an  artist 
is  creating  space  —  whether  physical,  iriNital.  visual 
or  emotional.  As  VE  technology  becomes  more 
widely  accessiUe,  it  is  reasonable  to  expect  that  artists 
win  begin  to  explore  the  unique  possibilities  for 
expression  that  virtual  environments  offer. 

8.  Communication  and  Collaboration 

Vinual  environments  coiqiled  with  hi^i-speed 
networks  and  distributed  computation  provide  a 
common  nemral  space  where  communkation, 
negotiation,  and  coUaboniion  can  take  place.  The 
ability  for  two  or  more  people  in  sqmrate  locations  to 
meet  inside  a  vinual  space  and  work  toward  a  common 
goal  has  the  potential  to  revolutionize  die  way 
collaborative  work  is  done. 


Needs 

To  date,  most  of  the  major  deficiencies  of  VE  have  been 
demonstrated  by  experimentation  with  various  applications. 
These  deficiencies  are  not  specific  to  the  applications  but  to 
VE  technology.  They  include: 

1.  Hardware,  software,  and  inierfoce  issues. 

These  have  been  discussed  in  detail  in  an  earlier  ponion 
of  the  report 

2.  Model  engineering. 

Maiy  researchers  have  found  that  building  a  reasonably 
detailed  model  for  a  non-trivial  objea  (i.e.,  a  house) 
takes  an  incfdinaie  amount  of  effort  (pethsq»  a  man- 
year).  The  need  for  improvement  is  obvious. 

3.  Psychtdogical  measurement  techniques  and  metrics. 

Measuring  the  possiMe  superiority  of  VE  techiKdogy 
ctften  involves  measuring  human  cognitive 
perfonnance.  This  type  of  measuiemem  is  extremely 
diSkalL 

4.  Simulator  sickness. 

Experience  with  space  flight  suggests  that  the  more 
realistic  VE  becomes  and  die  longer  peoftie  spend 
immersed  in  a  VE,  die  more  people  in  >^'s  may 
eiqierieoce  various  physical  reactions,  commonly 
known  as  simulator  sickness.  This  phenomenon  is 
also  dacusaed  in  an  egfier  portion  of  the  report 
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Recommendadons 

1.  Researchers  in  VE  should  look  toward  applications 

which  pnMDOte  measurement  NSF  slKwid  encourage 
applic^ons  which  provide  disoiminaiory  power  to 
ev^uate  VE  technology  versus  'through  the  window' 
inusactive  graphics  and  other  similar  technologies. 

2.  Wideqxead  development  t^VEa^^cations  are 

depetukm  on  the  availability  of  a  stable  and  robust 
software  infiastnicttiie.  NSF  should  address  the  issue 
of  making  such  a  software  environment  available  as 
new  devdqpors  enter  the  field.  The  alternative  is  to 
require  each  new  site  to  waste  valuable  nine 
"ronventing  the  wheel”. 

2.  NSF  funding  should  be  directed  toward  applications 

which  solve  real’world  problems.  VE  applications 
must  move  beyond  simple  demonstrations  of 
technology  and  begin  to  solve  problems  for  people 
where  they  are. 

3.  If  VE  applications  ate  to  be  taken  seriously,  the  causes 

and  efiects  of  sickness  from  virtual  environments  must 
be  well  understood.  To  this  end,  research  into 
'simulator  sickness”  should  be  supported.  VE  also 
seems  to  be  an  excellent  platform  for  related  and 
unrelated  psychological/^hysirdogical  measurements. 

4.  InterpeiscHial  Interaction  as  facilitated  by  VE.  Can  use 

of  VEs  to  communicate  between  pec^ie  approach  the 
level  of  communication  we  currently  experience  in 
person,  or  in  a  group?  NSF  funding  should  siqiport 
research  toward  resolving  these  questions. 

5.  Researchers  should  begin  work  on  the  probable  impaa 

of  VE  technology  on  socieiy:  Will  VEs  change  the 
way  we  weak  (telecommutingAeleconfeiencing)?  Will 
they  modify  our  interpersonal  interactians?  As  the 
lechnok^  becomes  more  readily  available,  how  will 
society  react? 
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Appendix 

Taxonomies  for  Virtual  Environments 

On  Taxonomies  In  General  Taxonomy  can  help 
thinking  about  a  group  of  ideas.  A  taxonomy  characterizes 
ideas  along  several  mote  or  less  independent  axes  so  as  to 
form  a  matrix.  The  taxonomy  has  two  important  parts,  the 
axes  and  the  entries  in  the  matrix. 

The  items  listed  on  each  axis  ate  related  by  whatever 
characteristic  is  represented  by  that  axis.  If  the  variable 
chosen  is  continuous,  the  axis  will  represent  a  spectrum  of 
values  which  may  be  more  or  less  well  defmed.  If  the 
variable  chosen  has  tmly  a  few  fixed  values,  then  those 
values  divide  the  space  of  all  ideas  considered  into 
categories.  For  example,  a  taxonomy  of  people  might 
include  eye  color  as  a  variable,  characterizing  eye  cokn*  into 
a  small  number  of  common  values,  e.g.  blue,  brown,  hazel, 
etc.  The  several  axes  of  the  taxonomy  divide  the 
intellectual  space  of  interest  into  a  number  of  boxes,  each 
of  which  can  be  examined  in  turn. 

The  boxes  in  a  taxonomy  are  used  to  enumerate  ideas  that 
have  common  characteristics  in  the  axes  chosen.  One  can 
list  in  each  such  box  a  set  of  ideas  that  ate  related  to  each 
other  by  their  position  on  the  axis.  If  ail  of  the  axes  ate 
continuous,  the  taxonomy  describes  a  space  of  possibilides. 
In  such  a  diagram,  each  related  collection  of  ideas  occupies  a 
region  of  the  overall  space  instead  of  a  discrete  box. 

The  number  of  boxes  in  a  taxonomy  is.  of  course,  its 
volume,  i.e.  the  product  of  the  number  of  values  permitted 


to  each  variable.  The  most  useful  taxonomies  generally 
have  fewer  than  1(X)  boxes.  With  more  boxes  it  is  difficult 
to  think  about  each  one  separately.  Some  useful 
taxonomies  have  only  a  few  boxes;  as  few  as  four  are 
sometimes  useful,  llie  desire  to  limit  the  number  of  boxes 
limits  the  number  of  values  permitted  to  each  variable.  Of 
course  if  there  are  more  variables,  each  must  be  permitted 
fewer  values.  A  useful  discrete  variable  usually  piovkles  2- 
5  values;  a  useful  continuous  variable  generally 
approximates  continuity  with  small,  medium,  and  large. 

There  ate  two  parts  of  building  a  taxonomy  that  make 
building  it  iiueiesdng.  Fim  there  is  the  selection  of  axes 
and  values  along  them.  In  forming  the  axes,  one  is  forced 
to  think  of  (»thogonal  dimensbns.  Which  variables  tue 
related  and  which  are  independent  It  often  happens  that  as 
the  taxonomy  takes  shape  one  finds  that  the  variaUes 
chosen  are  not  really  independent.  For  example,  in  fonning 
the  taxonomy  presented  here  we  initially  cho»  a  separate 
axis  to  distinguish  ’sensed  teal  world’  versus  ’simulated 
world.’  As  the  taxonomy  fcamed  we  considered  that  maybe 
"simulated’  could  be  included  as  a  single  value  along  the 
’time’ axis.  We  considered  making  this  change  because  we 
found  inadequate  distinction  between  boxes  differing  in  this 
dimension. 

The  second  interesting  part  of  building  a  taxonomy  is 
filling  in  the  boxes.  Here  the  really  interesting  results 
come  out  by  examining  combinations  of  variables  that  may 
hitherto  not  have  been  considered  together.  For  example, 
our  taxonomy  that  plots  different  sense  modalities  against 
different  physical  phenomena  leads  one  to  ask  what  smells 
look  like,  how  pictu!"ts  sound,  and  other  such  ideas  that 
might  not  otherwise  occur  to  unaided  thinking.  Making  a 
taxonomy  is  the  only  organized  way  I  know  to  go  about 
creative  thought 

Our  Taxonomies  for  Virtual  Environments  We 
actually  made  three  taxonomies,  which  divide  up  (he  space 
of  all  possible  virtual  environments  in  different  ways. 

BaaBotf-Bciaycgn  the  World  andHuman.  The  first 
taxonomy  comes  about  by  observing  that  a  mechanical  or 
electronic  device  can  be  interposed  b^een  a  human  being 
and  the  real  worid,  as  shown  in  Figure  1  below. 
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Figure  1.  Technologically-mediated 
experience. 

Inputs  and  outputs  from  the  real  wwld  and  the  human  are 
coinected  to  this  black  box.  This  diagram  covers  all  types 
of  technologically-mediated  experience,  in  which  some  sort 
of  apparatus  mediates  human  peicqxion  of  and  action 
within  the  world.  The  most  important  subcategories  of 
technologically-4nediated  experience  ate  recorded, 
transmitted,  and  simulated  experience.  Hgure  2  sh-  ws  that 
these  types  of  mediated  experience  ate  distinguiriied  by 
different  patterns  of  data  flow.  The  flow  for  a  robot  is 

also  shown  in  Ingure  2.  with  possible  supervision  by  a 
human  operator.  We  include  within  the  black  box  in  each 
of  the  diagrams  the  model  which  defines  the  virtual  world 
perceived  by  the  human. 


Simulated  Experience 


Robot 

(supervised  by  human) 


Figure  2.  Data  flow  for  types  of  mediated  experience. 


Two  or  more  people  may  communicate  or  interact  within  a 
virtual  environment,  which  leads  to  the  data  flow  diagrams 
in  Figure  3.  The  first  two  diagrams  emphasize  two  different 
things,  conununication  versus  interaction  with  the  virttal 
worid.  However  all  of  the  data  paths  exist  to  do  both 
simultaneously.  The  last  diagram  of  Figure  3  shows  that 
two  people  may  inieraa  collabocatively  with  the  real  worid 
tfaiough  a  shared  iqneseniation  of  the  real  world. 
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In  Real  Worid 

Via  Shared  Virtual  World 

Figures.  Two  people  in  shared  virtual  world 


Time  and  Space  Our  second  taxonomy  chooses  as  axes 
three  variables:  time,  space,  and  simulate  vs  reaL  In  this 
taxonomy,  the  dimensions  of  time  and  space  compare  the 
time  and  space  coordinates  of  the  real  world  as  captured  by  a 
sensor  versus  the  actual  physical  time-space  location  of  the 
human  immersed  in  a  vinu^  world.  For  example,  in 
teleoperadon,  the  space  coordinates  of  the  virtual  world 
perceived  by  the  human  are  different  from  the  human's  qace 
coordinates. 

We  have  characterized  both  time  and  space  as  "aligned", 
"displaced",  and  "scaled".  By  aligned  we  mean  in  real  time 
or  in  spatial  registry.  By  delayed  we  mean  passed  on 
through  time  by  some  kind  of  recording  or  di^laced 
through  space  as  in  one  of  the  many  "tele"  kinds  of  objects. 
By  scaled  we  mean  speeded  up,  slowed  down,  enlarged,  or 
minified.  Notice  that  if  scaled,  aligned  does  not  apply 
unless  two  scale  factors  happen  to  match. 

We  have  debated  how  to  treat  the  "simulated  versus  real" 
dimension.  One  of  us  claims  that  "simulated"  is  reaUy 


another  value  on  the  time  dimension  because  in  a 
simulation  "real  time"  is  a  matter  of  choice.  The  other 
claims  that  simulations  can  be  intended  to  model  physical 
systems  and  therefore  that  "real  time"  does  have  a  meaning 
within  a  simulation.  Entering  this  debate  is  instructive  in 
terms  of  generating  ideas.  For  this  report,  we  have  left 
"simulated  versus  real"  as  a  separate  dimension  of  the 
taxonomy. 

A  issue  that  comes  up  in  trying  to  classify  simulated 
virtual  worlds  is  that  while  some  simulations  are  intended 
to  model  the  real  world,  and  thus  can  be  classified  as  real¬ 
time  or  spatially  registered,  other  simulations  are 
completely  fanciful  and  don't  match  up  with  any  time  or 
place  in  the  real  w(»id.  Thus  "fantasy  worid  simulation" 
seems  to  be  a  separate  category  of  its  own  in  our 
taxonomy. 

We  have  drawn  the  taxonomy  hierarchically  in  two  stages. 
The  top  level  is  shown  in  Figure  4,  and  the  two  boxes 
"recording  or  transmission  of  real  worid"  and  "simuladon  of 
real  worid"  are  further  broken  down  in  Figure  S  along  the 
dimensions  of  relative  time  and  space.  The  entries  in  the 
boxes  of  Figure  S  are  examples,  not  category  defuiitions. 


Sense  Real  Simulate 
Worid _ 


Real  Worid 

recordingar 

simulation  of 

transmission  of 

real  world 

real  world 

Fabricated 

7 

simulation  of 

Worid 

fantasy  world 

Figure  4.  Top  level  taxonomy  of  vinual  environments. 
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Hgure  5.  Taxonomy  of  time,  space,  and  simulated  vs.  real. 


Sensors  versus  Human  Senses  Our  third  taxonomy  looks  at  extending  human  perception  within  and  across  sense 
modalities.  Here  one  axis  is  just  the  sensory  modalities.  The  other  axis  is  various  phenomena  that  could  be  sensed.  This 
matrix  is  shown  in  the  figure  below.  Neither  axis  has  a  complete  list  of  senses  nor  phenomeita. 
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Along  the  main  diagonal  of  this  matr?  are  augmentations 
and  prostheses  for  the  oniinary  ’’built-in '  human  senses. 

For  example,  spectacles  appear  in  the  visible  light  row 
under  vision,  a^  hearing  aids  are  listed  at  the  intersection 
of  sound  and  hearing.  The  diagonal  oitry  for  smell  prompts 
the  question  "Could  a  device  for  amplifying  smell  be 
cnsaiBd?" 

The  off-diagonal  boxes  would  contain  examples  of  sensory 
substitution.  For  example,  what  do  smells  look  like? 

What  do  sounds  look  lite?  How  do  sounds  smell?  It  is  the 
ability  of  this  taxonomy  to  generate  fresh  ideas  that  is 
interesting. 

The  lower  part  of  the  matrix  lists  phettomena  for  which  no 
corresponding  human  sensory  system  exists,  such  as 
ultrasound.  For  these  imperceptibie  phenomena,  mapping 
sensors  which  detect  them  to  human  senses  expands  human 
awareness  by  creating  "synthetic  senses." 

Isolated  versus  Merged  Another  distinction  not 
captured  in  any  of  the  earlier  taxonomies  in  this  report  is 
that  a  virtual  world  may  either  be  portrayed  to  the  human 
with  the  real  world  blacked  cot,  or  the  virtual  world  may  be 
superimposed  onto  the  human's  direct  perception  of  the  teal 
wt^d.  This  may  be  done  either  optically  with  half-silvered 
mirrors,  or  computatkmally  by  capturing  a  representation  of 
Che  real  world  with  cameias  and  then  mixing  the  oqxured 
model  of  the  real  world  with  another  model  representing  the 
virtual  world. 

It  is  most  useful  to  merge  the  real  and  virtual  wwids  when 
the  virtual  world  is  qiatially  registered  with  the  real  worid. 
However,  even  when  the  virtual  wmld  modds  a  distant 
locadon,  seeing  through  to  the  surrounding  real  world  may 
be  useful  simply  to  let  the  human  user  avoid  running  into 
walls  and  tripping  over  obstacles. 
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Abstract 

We  describe  initial  results  which  show  "live”  ultrasound 
echography  data  visualized  within  a  pregnant  human  subject.  The 
visualization  is  achieved  by  using  a  small  video  camera  mounted  in 
front  of  a  conventional  head-mounted  display  worn  by  an  observer. 
The  camera's  video  images  are  composited  with  computer-gener¬ 
ated  ones  that  contain  one  or  more  2D  ultrasound  images  properly 
transformed  to  the  observer’s  current  viewing  position.  As  the 
observer  walks  around  the  subject,  the  ultrasound  images  appear 
stationary  in  3-space  within  the  subject.  This  kind  of  enhancement 
of  the  observer’s  vision  may  have  many  other  applications,  e  g., 
image  guided  sutgical  procedures  and  on  location  3D  interactive 
architecture  preview 

CR  Categories;  1.3.7  Three-Dimensional  Graphics  and  Realism] 
Virtual  Reality,  1.3.1  [Hardware  architecture);  Three-dimensional 
displays.  1.3.6  [Methodology  and  Techniques]:  Interaction  tech¬ 
niques.  J.3  [Life  and  Medical  Sciences];  Medical  information  sys¬ 
tems. 

Additional  Keywords  and  Phrases:  Virtual  reality,  see-through 
head-mounted  display,  ultrasound  echography,  3D  medical  imaging 

1.  Introduction 

We  have  been  working  toward  an  ’ultimate’  3D  ultrasound 
system  which  acquires  and  displays  3D  volume  data  in  real  time. 
Real-time  display  can  be  crucial  for  applications  such  as  cardiac 
diagnosis  which  need  to  detect  certain  kinetic  features.  Our  ’ulti¬ 
mate’  system  design  requires  advances  in  both  3D  volume  data 
acquisition  and  3D  volume  data  display.  Our  collaborators,  Dr.  Olaf 
von  Ramm’s  group  at  Duke  University,  are  working  toward  real-time 
3D  volume  data  acquisition  [Smith  1991;  von  Ramm  1991].  At 
UNC-Chapel  Hill,  we  have  been  conducting  research  on  real-time 
3D  volume  data  visualization. 

Our  research  efforts  at  UNC  have  been  focused  in  three  areas; 
1 )  algorithms  for  acquiring  and  rendering  real-time  ultrasound  dau. 
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2)  creating  a  working  virtual  environment  which  acquires  and  dis¬ 
plays  3D  ultrasound  data  in  real  time,  and  3)  recovering  structural 
information  for  volume  rendering  specifically  from  ultrasound  data, 
which  has  unique  image  processing  requirements.  This  third  area  is 
presented  in  [Lin  1991  ]  and  is  not  covered  here. 

Section  2  of  this  paper  reviews  previous  work  in  3D  ultra¬ 
sound  and  Section  3  discusses  our  research  on  processing,  rendering, 
and  displaying  echographic  data  without  a  head-mounted  display. 
Since  the  only  real-time  volume  data  scanners  available  today  are  2D 
ultrasound  scanners,  we  try  to  approximate  our  'ultimate'  system  by 
incrementally  visualizing  a  3D  volume  dataset  reconstructed  from  a 
never-ending  sequence  of2Ddata  slices  (Ohbuchi  1990;  1991).  This 
is  difficult  because  the  volume  consisting  of  multiple  2D  slices  needs 
to  be  visualized  incrementally  as  the  2D  slices  are  acquired.  This 
incremental  method  has  been  successfully  used  in  off  line  experi¬ 
ments  with  a  3-dcgTee-of-frccdom  (DOF)  mechanical  arm  tracker 
and  is  extendible  to  6  degrees  of  freedom,  e.g.,  a  3D  translation  and 
a  3D  rotation,  at  greater  computational  cost. 

Sections  4  and  5  present  our  research  on  video  see-through 
head-mounted  display  (HMD)  techniques  involving  the  merging  of 
computer  generated  images  with  real-world  images.  Our  video  see- 
through  HMD  system  displays  ultrasound  echography  image  data  in 
the  context  of  real  (3D)  objects.  This  is  pan  of  our  continuing  see- 
through  HMD  research,  which  includes  both  optical  see-through 
HMD  and  video  see-through  HMD.  Even  though  we  concentrate 
here  on  medical  ultrasound  imaging,  applications  of  this  display 
technology  are  not  limited  to  it  (see  Section  6.2). 

2.  Previous  Research  in  3D  Ultrasound 

The  advantages  of  ultrasound  echography  are  that  it  is  rela¬ 
tively  safe  compared  with  other  imaging  modalities  and  that  images 
are  generated  in  real  time  [Wells  1977].  This  makes  it  the  preferred 
imaging  technique  for  fetal  examination,  cardiac  study,  and  guided 
surgical  procedures  such  as  fine-needle  aspiration  biopsy  of  breast 
tumors  [Fornage  1990].  Ultrasound  echography  offers  the  best  real¬ 
time  performance  in  3D  data  acquisition,  although  slower  imaging 
modalities  such  as  MRI  are  improving. 

The  drawbacks  of  ultrasound  imaging  include  a  low  signal  to 
noise  ratio  and  poor  spatial  resolution.  Ultrasound  images  exhibit 
“speckle"  which  appears  as  grainy  areas  in  images.  Speckle  arises 
from  coherent  sound  interference  effects  from  tissue  substructure. 
Information  such  as  blood  flow  can  be  derived  from  speckle  but  in 
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general  speckle  is  hard  lo  uiilue  (Thijsscn  !  v'iO]  Oihcr  prol'ierti\ 
with  ultrasound  imaging  include  aitenuation  that  incrtascs  with 
freguenc) .  phase  aberration  due  to  tissue  inhomogen  *>t> .  and  reilec 
non  and  refraction  artilacts  (Hams  199()j 

2.1  3D  L'ltrasound  Image  Acquisition 

Just  as  ultrasound  cchographs  has  evolved  from  ID  data 
acquisition  to  2D  data  acquisition,  work  is  in  progress  to  advance  to 
3D  data  acquisition.  Dr.  Olaf  von  Ramm' s  group  at  Duke  Umversitv 
is  developing  a  3D  scanner  which  will  acquire  3D  data  in  real  time 
[Shattuck  1984;Smith  1991;  von  Ramm  1991  j  The  3D  scanner  uses 
a  2D  phased  array  transducer  to  sweep  out  an  imaging  volume  A 
parallel  processing  technique  called  Expiosonan  is  used  on  return 
echoes  to  boost  the  data  acquisition  rate 

Since  such  a  rea'-ume  3D  medical  ultrasound  scanning  sys¬ 
tem  IS  not  yet  available,  pnor  studies  on  3D  ultrasound  imaging 
known  to  the  authors  have  tned  to  reconstruct  'O  data  from  imaging 
primitives  ofa  lesser  dimension  I  usually  2D  images)  To  reconstruct 
a  3D  image  from  images  of  a  lesser  dimeii,S(on.  the  location  and 
orientation  of  the  imaging  pnmuives  must  be  known  Coordinate 
vaiuesareexpliciiiy  tracked  either  acoustically  (Brinkley  I9'78.  King 
1990;  Moritz  1983).  mechanically  (Geiser  1982a.  Geiser  1982b. 
Hottier  1989;  McCann  1988.0hbuchi  1990;  Raichcicn  1986.  Sitckels 
19841,  or  optically  [Mills  1990]  In  other  systems,  a  human  or  a 
machine  makes  scans  at  predetermined  locations  and/or  oncniations 
(Collet  Billon  1990;  Ghosh  1982;  lioh  1979.  Laloochc  1989; 
Matsumoto  1981;  Nakamura  1984,  Tomographic  Technoksgies  1991 1 

A  particularly  interesting  system  underdevelopment  at  Philips 
Paris  Research  Laboratory  is  one  of  the  closest  yet  to  a  real-time  3D 
ultrasound  scanner  (Collet  Billon  19901.  It  is  a  follow  on  to  earlier 
work  which  featured  a  manually  guided  scanner  with  mechanical 
tracking  [Hottier  199'J!,  This  near  real-time  3D  scanner  is  a  mechani¬ 
cal  sector  scanner,  iti  which  a  conventional  2D  sector  scanhead  with 
an  annular  array  transducer  is  rotated  by  a  stepper  motor  to  get  a  third 
scanning  dimension.  In  a  penod  of  3  to  5  seconds,  50  to  100  slices 
of  2D  sector  scan  images  are  acquired.  Currently  the  annular  array 
tran.sducer  in  this  system  provides  better  spatial  resolution,  but  less 
temporal  resolution,  than  the  real-time  3D  phased  array  system  by 
von  Ramm  et  a!.,  mentioned  oixyve.  A  commercial  product,  the 
Echo-CT  system  by  Tomographic  Technologies,  GMBH,  uses  the 
linear  translation  of  a  transducer  inside  a  tube  inserted  into  the 
esophagus  to  acquire  parallel  slices  of  the  heart.  Image  acquisition 
is  gated  by  respiration  and  an  EKG  to  reduce  registration  problems 
(Tomographic  Technologies  1991]. 

2.2  3D  Ultrasound  Image  Display 

One  should  note  that  3D  image  data  can  be  presented  not  only 
in  visual  form,  but  also  as  a  set  of  calculated  values,  eg.,  a  ventncular 
volume.  The  visual  form  can  be  classified  further  by  the  rendering 
primitives  used,  which  can  be  either  geometnc  (e.g..  polygons)  or 
image-based  (e.g,,  vosel.s).  Many  early  studies  focu.scd  on  non- 
inva.sively  estimsiing  of  the  volume  of  the  heart  chamber  (Bnnkley 
1978:  Ghosh  1982;  Raichelen  1986;  Stickels  1984],  Typically.  2D 
echography  (2DF,)  images  were  stored  on  video  tape  and  manually 
processed  off-line.  Since  visual  presentation  wa.s  of  secondary 
interest,  wire  frames  or  a  slack  of  contours  were  often  used  to  render 
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KluJiev  rcconviruv ted  3D  grey  level  iitugr-.  pievcrving  grey  viaic 
which  cart  be  crucial  to  tissue  characierirain-tri  K  ollei  Biikm 
Hottier  19SV,  Laiouchc  Istgq.  McCaim  IWkg  Saiamuta  IdM  Ihiu 
lOSA).  Tomographic  Tcchnofogtes  jsbrij  (Lalouche  IdgVj  u,  * 
marrunogram  study  using  a  special  2DE  k anner  that  can  aa.quife  artd 
store  coosccuiivc  parallel  slices  ai  1  mni  inservafs  A  voiunie  is 
reconstructed  by  cubic -spiioe  imerpoUtion  and  then  volume  rrn 
dered  (McCarm  1988]  performed  gated  acquisdictn  of  a  bean  s 
image  over  a  cardias  eyefe  by  storing  2Dt  images  on  s  i«kt>  tape  and 
then  reconstructing  arxJ  volume  rendenng  thetti  Repetiiisr  low 
pass  filtering'  was  used  during  reconstruct  ton  lo  fill  the  spaces 
between  radial  slices,  which  suppressed  aliasing  artifacts 
(Tomographic  Technologies  1991  (pros  ides  firs  ible  re  slicing  by  up 
lo  6  planes  as  well  other  imaging  modes  (CoKci  Billon  199()j  uses 
Iwo  visualteaiioo  techniques  re- slicing  by  an  arbitrirv  plane  and 
volume  rendenng  The  lonner  alien* s  (aster  bui  only  2D  viewing  on 
a  current  worksiaiion  The  latter  allows  3D  viewing  hui  often 
involves  cumbervome  manual  segmenuiion  The  rrccmstructKin 
algonthm  uses  straightforward  k>w  pass  filtenng 

3.  iiKfcmental  Volume  VuiuUution 

We  has  been  espenmenimg  w-ith  volume  rrisdenng  av  one 
alternative  for  visuaiuing  dynamic  ultrasound  volume  dau  Stan 
dard  volume  rendering  techniques  which  rely  heavily  on  peeproiess 
ing  do  not  apply  well  todynamic  dau  which  must  he  visualized  in  real 
lime  (Levoy  1988,  Sabcili  1988,  Upson  1988)  We  review  here  a  i 
incremental  .  interact'  3D  ultrausound  visualization  technique 
which  visualizes  a  3D  volume  a.v  it  is  incrcmcnully  updated  by  a 
sequence  of  registered  2D  ultrasound  images  [Ohbuchi  1990,  199i| 

Our  target  function  is  sampled  at  irregular  points  and  may 
change  over  time  Instead  of  directly  visualizing  samples  from  this 
Ufgcl.  we  recoastruct  a  regular  3D  volume  from  this  time  senes  of 
spatially  irregular  sample  points  This  places  a  limit  on  storage  and 
computation  requirements  which  would  grow  without  bound  if  we 
reuined  all  the  past  sample  points  The  reconstructed  vrslumc  is  then 
rendered  with  an  incremcnul  volumc-rendcnng  technique 

The  reconstruction  is  a  4D  convolution  process  A  3D 
Gauvsian  kernel  is  used  for  spatial  reconstruction  followed  by  a 
temporal  reconstruction  based  on  simple  auto  regressive  moving 
average  (ARMA)  filtenng  [Haddad  1991],  Time  stamps  arc  as¬ 
signed  to  each  3D  voxel,  which  are  updated  dunng  reconstruction 
The  time  sump  difference  between  a  reconsmiaed  voxel  and  an 
incoming  sample  is  used  to  compute  coefficients  for  the  ARMA 
filter.  The  3DGau.ssian  filter  is  loosely  matched  to  the  point  spread 
function  of  the  ultrasound  transducer  and  is  a  good  choice  because  it 
minimizes  the  product  of  spatial  bandwidth  and  spatial  frequeiKy 
bandwidth  (Hildreth  1983.  Lcipnik  1960) 

An  image-order.  ray<asting  algonthm  based  on  (Levoy 
1988]  renders  the  final  images  incremcnully.  Rendenng  is  tnere- 
menul  and  fast  only  if  the  viewpoint  is  fixed  and  if  the  updated 
volume  is  relatively  small.  Shading  and  ray  sampling  are  done  only 
for  voxels  proximate  to  incoming  data.  The  ray  samples  are  stored 
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Figure  1.  Tvmi  ot  9()  2D  ulirasound  echograph>  images  of  a  plastic 
lov  doll  phantom  which  was  scanned  in  a  water  tank.  The  scans 
shown  are  at  the  torso  ( left  >  and  at  the  head  t  right ),  The  clouds  at  the 
bottom  of  the  scans  are  artifacts  due  to  reflections  from  the  bottom 
of  the  water  tank. 


Figure  2.  Reconstructed  and  rendered  image  of  the  toy  doll  phantom 
using  incremental  volume  visualization. 
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To  lest  this  s isualizaiion  technique  we  acquired  a  sene'  of 
2D  images  with  a  nianuails  guided  consentuma!  2DI  s  anhead 
attached  to  a  mechanical  tracking  arm  w  nh  t  DDF  i  two  iranslaiions 
and  one  rotation  i  As  w  e  scanned  s  ariuus  laigeis  n  a  w  utcr  lank  iheir 
images  and  ih: or  corresponding  geomeirs  w  err  stored  of!  lim"  V>  r 
then  ran  the  incremental  volume  sisualizaiion  slgonthm  on  a 
DECstaiion  .‘itXXi  with  256  MB  iif  memors  using  tho  data  W  ah  a 
reconstruction  buffer  si/c  ot  !  6(i  >  i  sp  >  t(Ki  and  an  image  si/e  o( 
256  *  25b.  It  look  15-20  seconds  to  rcionsiruct  and  rendei  a  ivpu  ai 
image  alter  insertion  of  a  2D  data  slice  This  nme  varied  ssihi 
reconstmciion.  shading,  and  viewing  parameters 

Figure  1  shows  2  out  of  90  2D  images  ot  a  pi  tic  lov  doll 
phantom  which  is  visualized  in  Figure  2  The  2D  images  were 
produced  bv  an  ATI.  .Mark -I  Scanner  w  itha  5  5  MHz  linear  sc  anhead 
The  2D  images  overlap  but  are  roughh  parallel  af  apprmimaieis 
2  mm  inicrvals 


4.  Virtual  Environment  l  ltrasound  imaging 

Various  medical  ultrasound  imaging  applications  require  a 
registration  of  ultrasound  images  w  ith  anaiomicai  references,  e  g  .  in 
performing  a  r nc  needle  aspiraiion  hiopss  ol  a  suspect. ’d  breast 
lumor|Fomagc  |W0]  A  virtual  environment  whichdisplays  images 
acquired  by  ultrasound  equipment  in  plat  c  w  iihin  a  patient  s  anatoms 
could  facihiatc  such  an  application  \k  c  have  developed  an  etpen- 
menial  system  that  displays  muliipie  2D  medical  ultrasound  images 
overlaid  on  real-world  images  In  January  1992.  alter  months  of 
development  with  test  objects  in  water  tanks,  wo  perfom»<-.'  vur  first 
experiment  with  a  human  subject 

Our  virtual  environment  ultrasound  imaging  system  works  as 
follows  (note  that  this  is  a  different  system  than  our  older  one 
described  in  the  previous  section  i  as  each  echography  image  is 
acquired  by  an  ultrasound  scanner,  its  position  and  oneniaiion  m  .^D 
world  space  are  tracked  with  6  degrees  of  freedom  i  [X)F  i  .Simuiia- 
neousiy  the  position  and  oncniaiion  of  a  HMD  arc  also  tracked  w  iih 
6  DOF  Using  this  geometry .  an  image-generation  sysicm  generates 
.^D  renderings  of  the  2D  ultrasound  images  These  images  are  video 
mixed  w  iih  real-world  images  from  a  miniature  TV  camera  mounted 
on  the  HMD  The  resulting  compiosite  image  shows  the  2D  ultra¬ 
sound  data  registered  in  its  true  5D  location 

Figure  is  a  block  diagram  of  our  system's  hardware  There 
are  three  major  components  1 1  an  imagc-acquisiiion  and  tracking 
system,  which  consists  of  an  ultrasound  scanner  and  a  Polhemus 
tracking  system.  2)  an  image-generation  system,  which  is  our  Pixel- 
Planes  5  graphics  multicomputer,  and  .U  a  HMD  which  includes  a 
ponabic  TV  camera,  a  video  mixer,  and  a  VPL  EyePhone  Each 
component  is  described  in  more  detail  m  Sections  4  1-4  5. 
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Figure  3.  Hardware  block  diagram  for  the  virtual  environment 
ultrasound  system. 


4.1  Image  Acquisition  and  Tracking 

Two  dimensional  uiv.asound  images  are  generated  by  an 
IREX  System  III  echography  scanner  with  a  16  mm  aperture 
2.5  MHz  phased  array  transducer.  These  images  are  digitized  by  a 
SUN  4  with  a  Matrox  MVP/S  real-time  video  digitizer  and  trans¬ 
ferred  to  our  Pixel-Planes  5  graphics  multicomputer  [Fuchs  19891. 
The  SU.N  4  operates  as  a  2DE  image  server  for  requests  from  the 
Pixel-Planes  5  system.  Images  are  distributed  among  the  Graphics 
Processors  (GPs)  on  a  round-robin  .scan-line  by  scan-line  basis.  Due 
to  the  bandwidth  limitations  of  the  SUN  4  VME  bus,  transfer  of  the 
512  X  480  X  8  bits/pixel  images  is  limited  to  2  Hz. 

A  Polhemus  system  with  one  source  and  two  receivers  is  used 
for  tracking  fPolhemus  1980].  One  receiver  tracks  the  HMD.  The 
other  tracks  the  ultrasound  transducer.  The  Polhemus  system  is 
mounted  in  non  ferrous  materials  away  from  magnetic  interference 
.sources  such  as  the  ultrasound  transducer.  HMD.  and  other  lab 
equipment.  A  calibration  procedure  is  used  to  relate  both  the 
ultrasound  transducer  to  its  Polhemus  receiver  and  the  HMD  TV 
camera  to  its  Polhemus  receiver  mounted  on  the  HMD.  This 
calibration  procedure  is  described  in  Section  4.4. 

4.2  Image  Generation 


eniiriinment  arc  rcci'icrcd  lo  ibc  real  world  wiihm  ihc  uf'J.iic-raic 
iiinit  ol  the  tracking  and  displ.o  s',  sicni  and  lun  w  iihm  the  a..qu!M. 
iion-ratc  liniii  oi  the  inia;ac-.icqui'iiK'ii  ''stem 

Pivcls  Irom  the  2D  ultra-i'und  miagC'.  arc  rendered  a-  ^null 
unshaded  sphere  pririiiliies  in  the  iinuat  ein  iromiieni  The  2D 
ultrasound  images  appear  as  spaec-fillinc  shcc'  recisiered  in  iiieii 
correct  .^D  position  The  ultrasound  images  are  distnbulcd  a.hiong 
the  GPs  where  thes  are  clipped  to  remove  unnecessary  margins  and 
transformed  into  sphere  primitives,  which  are  then  scni  lo  the 
Renderer  boards  for  direct  rasterization.  Pixel-Planes  5  renders 
spheres  very  rapidly,  even  faster  than  it  renders  mangles,  over  2 
million  per  second  (Fuchs  1985:  1989).  Final  images  are  assembled 
in  double  buffered  NTSC  frame  buffers  for  display  on  the  H.MD  To 
reduce  the  number  of  sphere  pnmiiives  displayed,  the  ultrasound 
images  are  filtered  and  subsampled  at  every  4ih  pixel.  Due  to  the  low 
resolution  of  the  HMD  and  inherent  bandwidth  iimiiaiion  of  the 
ultrasound  scanner,  this  subsampling  does  not  result  in  a  substaniial 
loss  of  image  quality.  An  option  to  threshold  lower  intensity  pixels 
in  2D  ultrasound  images  prior  to  3D  rendering  can  suppress  lower 
intensity  pixels  from  being  displayed. 

4J  Video  Sec-Through  H.MD 

A  video  sce-ihrough  HMD  system  combines  real-world  im¬ 
ages  captured  by  head-mounted  TV  cameras  with  synthetic  images 
generated  to  correspond  with  the  real-world  images.  The  important 
issues  are  tracking  the  real-world  cameras  accurately  and  generating 
the  correct  synthetic  images  to  model  the  views  of  the  cameras 
Correct  stereo  modeling  adds  concerns  about  matching  a  pair  of 
cameras  to  each  other  as  well  as  tracking  and  modeling  them. 
[Robineii  1991)  discusses  stereo  HMD  in  detail  and  includes  an 
analysis  of  the  VPL  EyePhone 

A  Panasonic  GP-KS102  camera  provides  monocular  see- 
through  capability  for  the  left  eye  in  our  current  system.  Images  from 
thiscameraare  mixed  with  synthetic  images  from  the  Pixel-Planes  5 
system  using  the  luminance  (bnghiness)  keying  feature  on  a  Grass 
Valley  Group  Model  100  video  mixer.  With  luminance  keying,  the 
pixels  in  the  output  image  arc  selected  from  either  the  real-world 
image  or  the  synthetic  image,  depending  on  the  luminance  of  pixels 
in  the  synthetic  image.  The  combined  image  for  the  left  eye  and  a 
synthetic  image  only  for  the  right  eye  are  dr  .  .  .  on  a  VPL 
EyePhone. 

4.4  Calibration 

Two  transformations,  a  "transducer  transformation"  and  a 
"camera  transformation.”  arc  needed  to  calibrate  our  test  system. 
The  transducer  transformation  relates  the  position  and  onentaiion  of 
the  Polhemus  tracker  attached  to  the  ultrasound  transducer  to  the 
position  and  scale  of  2D  ultrasound  image  pixels  in  3D  space.  The 
camera  transformation  relates  the  position  and  orientation  of  the 
head-mounted  Polhemus  tracker  to  the  HMD  TV  camera  position, 
orientation,  and  field  of  view. 


Images  are  generated  by  the  Pixel-Planes  5  system  based  on 
geometry  information  from  the  tracking  system.  Pixel-Planes  5  runs 
a  custom  PHIGS  implementation  which  incorporates  a  facility  to 
update  display  structures  asynchronously  from  the  display  process. 
This  separates  the  interactive  virtual  environment  update  rate  from 
the  2D  ultrasound  image  data  acquisition  rate.  Images  in  the  virtual 


Both  transformations  are  calculated  by  first  locating  a  cali¬ 
bration  jig  in  both  the  lab  (real)  and  tracker  (virtuall  3D  coordinate 
systems.  This  is  accomplished  by  performing  rigid  body  rotations 
with  the  transducer  tracker  about  axes  which  are  to  be  fixed  in  both 
the  real  and  virtual  coordinate  systems.  Two  samples  from  the 
tracker,  each  con.sisting  of  both  a  position  and  an  orientation,  are 
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sut  i  iciem  lo  l'i\  each  calibration  aMs.  The  transducer  transformation 
Is  computed  h>  lakins;  an  ultrasound  imape  ol  a  target  of  known 
geomeirc  placed  at  a  known  position  on  the  calibration  jic.  Bs 
finding  the  pixel  coordinates  of  point  targets  in  the  ultrasound  image, 
the  w  orld  coordinates  of  pixels  in  the  ultrasound  image  can  be  found 
From  this  relationship  and  the  location  of  the  Polhemus  tracker 
attached  to  the  ultrasound  transducer  at  the  time  the  target  was 
imaged,  the  transducer  tran.sformaiion  is  derived.  Similarly,  the 
camera  transformation  is  found  by  placing  the  HMD  TV  camera  at 
known  positions  and  orientations  relative  to  the  calibration  jig.  The 
field  of  view  of  the  TV  camera  i.s  known  from  camera  specifications. 
Manual  adjustments  are  used  to  improve  the  camera  transformation. 

S.  Experimental  Results 

In  January  1992  we  conducted  an  experiment  with  a  live 
human  subject  using  the  method  described  above.  We  scanned  the 
abdomen  of  a  volunteer  who  was  38  weeks  pregnant.  An  ultrasound 
technician  from  the  Depanment  of  Obstetrics  &  Gynecology  of  the 
UNC  Hospitals  performed  the  ultrasound  scanning. 

Figure  4  is  a  scene  from  the  experiment.  A  person  looks  on 
with  modified  VPL  EyePhone  with  the  miniature  video  camera 
mounted  on  top  and  in  front.  Figure  5  shows  the  left  eye  view  from 
the  HMD.  a  composition  of  synthetic  and  real  images.  Figure  6  is 
another  view  from  the  left  eye  of  the  HMD  wearer  which  shows 
several  2D  ultrasound  images  in  place  within  the  subject’s  abdomen. 


Figure  5.  A  video  image  presented  to  the  left  eye  of  the  HMD 
showing  a  view  of  the  subject ’s  abdomen  with  a  2D  ultrasound  image 
superimposed  and  registered.  Note  the  ultrasound  transducer  regis¬ 
tered  with  the  image  acquired  by  it.  The  2D  image  is  from  the  antero¬ 
inferior  view. 
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6.  Conclusions  and  Future  Directions 

The  resuIlN  presemed  >o  lar  are  ihe  initial  siepv  in  the  t'lrM 
application  ol  \shai  we  hope  will  be  a  llounshmi:  area  ot  computer 
graphics  and  visualization. 

6.1  Remaining  Technical  Problems 

1)  Conflicting  visual  cues:  Our  experiment  (Figures  5  and  6) 
showed  that  simply  overlaying  synthetic  images  on  real  ones  is  not 
sufficient.  To  the  user,  the  ultrasound  images  did  not  appear  to  be 

the  subject,  so  much  as  pasted  Or  wpofher.  To  overcome  this 
problem,  we  now  provide  additional  ■  es  to  the  user  by  making  a 
virtual  hole  in  the  subject  (Figure  by  digitizing  points  on  the 
abdomenal  surface  and  constructing  a  shaded  polygonal  pit.  The  pit 
provides  occlusion  cues  by  obscuring  the  abdomenal  surface  along 
the  inside  walls  of  the  pit.  Shading  the  pit  provides  an  additional  cue. 
Unfortunately,  this  does  not  completely  solve  (he  problem:  the  pit 
hides  everything  in  the  real  image  that  is  in  the  same  location  ( in  2D  I 
as  the  pit.  including  real  objects  that  are  closer  in  3D  than  the  pit. 
(Note  in  Figure  7,  the  edge  of  the  transducer  is  hidden  behind  the  pit 
representation  even  though  it  should  appear  in  front  of  it.) 

To  solve  this  problem,  the  systems  needs  to  know  depth 
information  for  both  the  real  and  synthetic  objects  visible  from  the 
HMD  user's  viewpoint.  This  would  make  it  possible  to  present 
correct  occlusion  cues  by  combining  the  live  and  synthetic  images 
with  a  Z-buffer  like  algorithm.  An  ideal  implementation  of  this 


Figure  7.  An  image  showing  a  synthetic  hole  rendered  around 
ultrasound  image.s  in  an  attempt  to  avoid  conflicting  visual  cues. 
Note  the  depth  cues  provided  by  occlusion  of  the  image  slices  by  the 
pit  walls  and  shading  of  Ihe  pit.  Also  note  the  incorrect  obscuration 
of  Ihe  ultrasound  transducer  by  the  pit  wall.  (RT32(X)  Advantage  II 
ultrasound  scanner  courtesy  of  Genera!  Electric  Medical  Systems.) 


would  require  re.il-linie  ranee  linjine  in'in  ilie  \tev\piiin\  oi  ihe 
HMDuxer  -  .iMenilieani  leetiiiiej!  elulienee  Ciraptnes  .irehiieeiui  ' 
that  proside  re.il-nnie  depili-ha-ed  imaee  ei>nipt."ilio!i  .iie  alieacl' 
under  deselopmcni  (Moinar  10021 

Another  rcmainme  problem  is  the  s  isuaii/aiionol  iniema!  3D 
structure  in  data  captured  b\  the  ultrasound  scanner  Neither  our 
incremental  volume  rendering  algorithm  (Scciion  3i  nur  multiple 
explicit  image  slices  in  .'-space  (Figure  bi  solve  this  problem  well  A 
combination  of  muiiipic  visualization  methods  will  probably  be 
necessary  in  the  future.  We  suspect  that  this  problem  is  difficult 
because  the  human  visual  system  is  not  accustomed  to  seeing 
structure  within  opaque  objects,  and  so  our  development  cannot  be 
guided  b>  the  "gold  standard"  of  reality  that  has  been  used  so 
effectively  in  guiding  other  3D  rendering  investigations. 

2)  System  lag:  Lag  in  image  generation  and  tracking  is  noticeable 
in  all  head-mounted  displays;  but  it  is  dramatically  accentuated  w  ith 
see-through  HMD.  The  “live  video"  of  the  observer’s  surroundings 
moves  appropriately  during  any  head  movement  but  the  synthetic 
image  overlay  lags  behind.  This  is  currently  one  of  our  system's 
major  problems  which  prevents  it  from  giving  the  user  a  convincing 
experience  of  seeing  synthetic  objects  or  images  hanging  in  3-space . 
A  possible  .solution  may  be  to  delay  Ihe  live  video  images  so  that  their 
delay  matches  that  of  the  synthetic  images.  This  w  ill  align  the  real 
and  synthetic  images,  but  won’t  eliminate  the  lag  itself.  W'e  are  also 
considering  predictive  tracking  as  a  way  to  reduce  the  effect  of  the  lag 
[Liang  19911.  Developers  of  some  multi-million  dollar  flight  simu¬ 
lators  have  .studied  predictive  trackingfor  many  years,  but  unfortu¬ 
nately  for  us.  they  have  not.  to  our  knowledge,  published  details  of 
their  methods  and  their  methods'  effectiveness  For  the  immediate 
future,  we  are  planning  to  move  to  our  locally-developed  "ceiling 
tracker"  [Ward  1992)  and  use  predictive  tracking. 

3)  Tracking  system  range  and  stability:  Even  though  we  are  using 
the  most  popular  and  probably  most  effective  commercially  avail¬ 
able  tracking  system  from  Polhemus.  we  are  constantly  plagued  by 
limitations  in  tracking  volume  and  tracking  stability  [Liang  1991  j. 
The  observer  often  steps  inadvertently  out  of  tracker  range,  and  even 
while  keeping  very  still  the  observer  must  cope  with  objects  in  the 
synthetic  image  “swimming"  in  place.  We  are  eagerly  awaiting  the 
next  generation  of  tracking  systems  from  Polhemus  and  other  manu¬ 
facturers  that  arc  said  to  overcomemosi  of  these  problems.  Even 
more  capable  tracking  sy.siems  will  be  needed  in  order  to  satisfy  the 
many  applications  in  which  the  observer  must  move  about  in  the  real 
world  instead  of  a  laboratory,  operating  room  or  other  controlled 
environment.  Many  schemes  have  been  casually  proposed  over  the 
years,  but  we  know  of  no  dev  ice  that  has  been  bui  It  and  demonstrated . 
Even  the  room-size  tracker  we  built  and  demonstrated  for  a  week  at 
SlGGRAPH’9i  still  needs  special  ceiling  panels  with  infrared  LEDs 
[Ward  1992). 

4)  Head-mounted  display  system  resolution:  For  many  of  the 
applications  envisioned,  the  image  quality  of  current  head-mounted 
video  displays  is  totally  inadequate.  In  a  see-through  application,  a 
user  is  even  more  sensitive  to  the  limitations  of  his  head-mounted 
display  than  in  a  conventional  non-see-through  application  because 
he  is  painfully  aware  of  the  visual  details  he’s  missing, 

5)  More  powerful  display  engines;  Even  with  all  the  above 
problems  solved,  the  synthetic  images  we  would  like  to  see.  for 
example,  real-time  volume  visualization  of  real-time  volume  data, 
would  still  take  too  long  to  be  created.  Much  more  powerful  image 
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generation  systems  are  needed  if  we  are  to  be  able  to  visualize 
usefully  detailed  3D  imagery  . 

Other  Applications 

1 )  Vision  in  surgery:  In  neurosurgery,  ultrasound  is  already  used  to 
image  nearby  arteries  that  should  be  avoided  by  an  impending 
surgical  incision. 

2)  Burning  buildings:  With  close-range,  millimeter  wavelength 
radar,  rescuers  may  be  able  to  “see  through”  the  smoke  in  the  interior 
of  burning  buildings. 

3)  Building  geometry:  Geometry  or  other  structural  data  could  be 
addod  to  a  “live"  scene.  In  the  above  "burning  building”  scenario, 
pans  of  a  building  plan  could  be  superimposed  onto  the  visual  scene, 
such  as  the  location  of  stairways,  hallways,  or  the  best  exits  out  of  the 
building. 

4)  Service  information:  Information  could  be  displayed  to  a  service 
technician  working  on  complicated  machinery  such  as  a  jet  engine. 
Even  simpler  head-mounted  displays,  ones  without  head  tracking, 
already  provide  information  to  users  on  site  and  avoid  using  a  large 
cumbersome  video  screens.  Adding  head  tracking  would  allow  3D 
superimposition  to  show,  for  instance,  the  location  of  special  parts 
within  an  engine,  or  the  easiest  path  for  removal  or  insertion  of  a 
subassembly. 

5)  Architecture  on  site:  Portable  systems  could  allow  builders  and 
architects  to  preview  buildings  on  site  before  construction  or  visual¬ 
ize  additions  to  existing  architecture. 

With  the  work  presented  here  and  the  identification  of  prob¬ 
lems  and  possibilities  for  further  research,  we  hope  to  encourage 
applications  not  only  of  “virtual  environments”  (imaginary  worlds), 
but  also  applications  that  involve  an  “enhancement  of  vision”  in  our 
real  world. 
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